Attachment 885644 Details for Bug 1086951 – VDSM Log

VDSM Log

vdsm.log (text/plain), 2.11 MB, created by Maurice James on 2014-04-12 03:12:08 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Maurice James

Created: 2014-04-12 03:12:08 UTC

Size: 2.11 MB

patch

obsolete

>Thread-22::DEBUG::2014-04-11 22:01:01,888::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:01:01,910::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000340689 s, 1.1 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:02,105::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:02,106::task::595::TaskManager.Task::(_updateState) Task=`d5fe3e2f-b47c-4bce-a990-d0e03035d5af`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:02,106::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:02,106::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:02,107::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:02,107::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:02,107::task::1185::TaskManager.Task::(prepare) Task=`d5fe3e2f-b47c-4bce-a990-d0e03035d5af`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:02,108::task::595::TaskManager.Task::(_updateState) Task=`d5fe3e2f-b47c-4bce-a990-d0e03035d5af`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:02,108::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:02,108::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:02,109::task::990::TaskManager.Task::(_decref) Task=`d5fe3e2f-b47c-4bce-a990-d0e03035d5af`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:03,113::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:03,141::task::595::TaskManager.Task::(_updateState) Task=`60f26a97-6c95-45cc-888e-d3f1ec3b9a8b`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:03,142::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:03,142::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:03,142::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:03,143::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:03,143::task::1185::TaskManager.Task::(prepare) Task=`60f26a97-6c95-45cc-888e-d3f1ec3b9a8b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:03,143::task::595::TaskManager.Task::(_updateState) Task=`60f26a97-6c95-45cc-888e-d3f1ec3b9a8b`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:03,144::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:03,144::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:03,144::task::990::TaskManager.Task::(_decref) Task=`60f26a97-6c95-45cc-888e-d3f1ec3b9a8b`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:01:03,395::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:01:03,407::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000293048 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:04,149::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:04,150::task::595::TaskManager.Task::(_updateState) Task=`3c4d349a-5389-41d7-a189-fc599da780ef`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:04,150::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:04,151::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:04,151::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:04,151::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:04,152::task::1185::TaskManager.Task::(prepare) Task=`3c4d349a-5389-41d7-a189-fc599da780ef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:04,152::task::595::TaskManager.Task::(_updateState) Task=`3c4d349a-5389-41d7-a189-fc599da780ef`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:04,153::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:04,153::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:04,154::task::990::TaskManager.Task::(_decref) Task=`3c4d349a-5389-41d7-a189-fc599da780ef`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:04,961::task::595::TaskManager.Task::(_updateState) Task=`3c94ac1d-e613-461d-8597-3213799f8883`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:04,961::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-13::INFO::2014-04-11 22:01:04,962::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '33.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340689', 'lastCheck': '3.0', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.3', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000565291', 'lastCheck': '5.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.3', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '12.0', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000293048', 'lastCheck': '1.5', 'valid': True}}
>Thread-13::DEBUG::2014-04-11 22:01:04,963::task::1185::TaskManager.Task::(prepare) Task=`3c94ac1d-e613-461d-8597-3213799f8883`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '33.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340689', 'lastCheck': '3.0', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.3', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000565291', 'lastCheck': '5.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.3', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '12.0', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000293048', 'lastCheck': '1.5', 'valid': True}}
>Thread-13::DEBUG::2014-04-11 22:01:04,963::task::595::TaskManager.Task::(_updateState) Task=`3c94ac1d-e613-461d-8597-3213799f8883`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:04,963::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:04,964::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:04,964::task::990::TaskManager.Task::(_decref) Task=`3c94ac1d-e613-461d-8597-3213799f8883`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:05,158::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:05,158::task::595::TaskManager.Task::(_updateState) Task=`b04d6d8c-db82-48ef-9fda-034326fe277e`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:05,159::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:05,159::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:05,160::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:05,160::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:05,161::task::1185::TaskManager.Task::(prepare) Task=`b04d6d8c-db82-48ef-9fda-034326fe277e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:05,161::task::595::TaskManager.Task::(_updateState) Task=`b04d6d8c-db82-48ef-9fda-034326fe277e`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:05,162::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:05,162::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:05,162::task::990::TaskManager.Task::(_decref) Task=`b04d6d8c-db82-48ef-9fda-034326fe277e`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:06,167::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:06,168::task::595::TaskManager.Task::(_updateState) Task=`fd6437d2-54c1-4603-b4e1-85c37e05b865`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:06,169::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:06,169::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:06,170::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:06,170::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:06,170::task::1185::TaskManager.Task::(prepare) Task=`fd6437d2-54c1-4603-b4e1-85c37e05b865`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:06,171::task::595::TaskManager.Task::(_updateState) Task=`fd6437d2-54c1-4603-b4e1-85c37e05b865`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:06,171::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:06,171::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:06,171::task::990::TaskManager.Task::(_decref) Task=`fd6437d2-54c1-4603-b4e1-85c37e05b865`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:07,176::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:07,177::task::595::TaskManager.Task::(_updateState) Task=`c3f2b873-94bc-40e4-ae78-a990b29f8925`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:07,177::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:07,178::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:07,178::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:07,178::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:07,178::task::1185::TaskManager.Task::(prepare) Task=`c3f2b873-94bc-40e4-ae78-a990b29f8925`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:07,179::task::595::TaskManager.Task::(_updateState) Task=`c3f2b873-94bc-40e4-ae78-a990b29f8925`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:07,179::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:07,179::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:07,180::task::990::TaskManager.Task::(_decref) Task=`c3f2b873-94bc-40e4-ae78-a990b29f8925`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:08,183::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:08,184::task::595::TaskManager.Task::(_updateState) Task=`a99a7389-c991-4613-9c5b-e3fda4746ab0`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:08,185::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:08,185::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:08,185::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:08,186::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:08,186::task::1185::TaskManager.Task::(prepare) Task=`a99a7389-c991-4613-9c5b-e3fda4746ab0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:08,186::task::595::TaskManager.Task::(_updateState) Task=`a99a7389-c991-4613-9c5b-e3fda4746ab0`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:08,186::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:08,187::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:08,187::task::990::TaskManager.Task::(_decref) Task=`a99a7389-c991-4613-9c5b-e3fda4746ab0`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:09,191::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:09,192::task::595::TaskManager.Task::(_updateState) Task=`e33f762b-dc4a-4768-928e-141b364c5a9d`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:09,192::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:09,192::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:09,193::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:09,193::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:09,193::task::1185::TaskManager.Task::(prepare) Task=`e33f762b-dc4a-4768-928e-141b364c5a9d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:09,194::task::595::TaskManager.Task::(_updateState) Task=`e33f762b-dc4a-4768-928e-141b364c5a9d`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:09,194::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:09,194::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:09,194::task::990::TaskManager.Task::(_decref) Task=`e33f762b-dc4a-4768-928e-141b364c5a9d`::ref 0 aborting False
>Thread-25::DEBUG::2014-04-11 22:01:09,808::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-25::DEBUG::2014-04-11 22:01:09,823::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000546938 s, 1.4 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:10,200::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:10,201::task::595::TaskManager.Task::(_updateState) Task=`faec843c-6b99-4781-9636-93697727aed9`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:10,201::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:10,201::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:10,202::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:10,202::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:10,203::task::1185::TaskManager.Task::(prepare) Task=`faec843c-6b99-4781-9636-93697727aed9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:10,203::task::595::TaskManager.Task::(_updateState) Task=`faec843c-6b99-4781-9636-93697727aed9`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:10,204::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:10,204::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:10,204::task::990::TaskManager.Task::(_decref) Task=`faec843c-6b99-4781-9636-93697727aed9`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:11,209::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:11,210::task::595::TaskManager.Task::(_updateState) Task=`5cd13cef-032b-4357-843f-b5e75ed6d160`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:11,210::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:11,211::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:11,211::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:11,211::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:11,212::task::1185::TaskManager.Task::(prepare) Task=`5cd13cef-032b-4357-843f-b5e75ed6d160`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:11,212::task::595::TaskManager.Task::(_updateState) Task=`5cd13cef-032b-4357-843f-b5e75ed6d160`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:11,212::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:11,213::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:11,214::task::990::TaskManager.Task::(_decref) Task=`5cd13cef-032b-4357-843f-b5e75ed6d160`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:01:11,946::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:01:11,961::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000337815 s, 1.1 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:12,257::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:12,258::task::595::TaskManager.Task::(_updateState) Task=`1a1642e5-2a6c-465a-8c58-65c1178aa44f`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:12,259::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:12,259::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:12,260::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:12,260::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:12,260::task::1185::TaskManager.Task::(prepare) Task=`1a1642e5-2a6c-465a-8c58-65c1178aa44f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:12,261::task::595::TaskManager.Task::(_updateState) Task=`1a1642e5-2a6c-465a-8c58-65c1178aa44f`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:12,261::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:12,261::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:12,261::task::990::TaskManager.Task::(_decref) Task=`1a1642e5-2a6c-465a-8c58-65c1178aa44f`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:13,266::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:13,269::task::595::TaskManager.Task::(_updateState) Task=`18062adc-e410-449a-a197-8f65ec3661d2`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:13,269::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:13,270::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:13,270::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:13,270::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:13,271::task::1185::TaskManager.Task::(prepare) Task=`18062adc-e410-449a-a197-8f65ec3661d2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:13,271::task::595::TaskManager.Task::(_updateState) Task=`18062adc-e410-449a-a197-8f65ec3661d2`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:13,271::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:13,271::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:13,272::task::990::TaskManager.Task::(_decref) Task=`18062adc-e410-449a-a197-8f65ec3661d2`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:01:13,434::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:01:13,449::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000239907 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:14,276::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:14,277::task::595::TaskManager.Task::(_updateState) Task=`1e9bea24-d58e-4f1d-859b-0b0919f315d6`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:14,278::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:14,278::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:14,279::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:14,279::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:14,280::task::1185::TaskManager.Task::(prepare) Task=`1e9bea24-d58e-4f1d-859b-0b0919f315d6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:14,280::task::595::TaskManager.Task::(_updateState) Task=`1e9bea24-d58e-4f1d-859b-0b0919f315d6`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:14,281::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:14,281::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:14,281::task::990::TaskManager.Task::(_decref) Task=`1e9bea24-d58e-4f1d-859b-0b0919f315d6`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:15,285::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:15,286::task::595::TaskManager.Task::(_updateState) Task=`cfbb1c02-5c4c-42a1-810b-67e802ecc3c9`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:15,286::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:15,287::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:15,287::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:15,287::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:15,288::task::1185::TaskManager.Task::(prepare) Task=`cfbb1c02-5c4c-42a1-810b-67e802ecc3c9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:15,288::task::595::TaskManager.Task::(_updateState) Task=`cfbb1c02-5c4c-42a1-810b-67e802ecc3c9`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:15,288::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:15,289::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:15,289::task::990::TaskManager.Task::(_decref) Task=`cfbb1c02-5c4c-42a1-810b-67e802ecc3c9`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:16,294::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:16,295::task::595::TaskManager.Task::(_updateState) Task=`da9cbcdb-c143-427b-84e0-cc8012faa320`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:16,296::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:16,296::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:16,297::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:16,297::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:16,297::task::1185::TaskManager.Task::(prepare) Task=`da9cbcdb-c143-427b-84e0-cc8012faa320`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:16,298::task::595::TaskManager.Task::(_updateState) Task=`da9cbcdb-c143-427b-84e0-cc8012faa320`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:16,298::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:16,299::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:16,299::task::990::TaskManager.Task::(_decref) Task=`da9cbcdb-c143-427b-84e0-cc8012faa320`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:17,304::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:17,304::task::595::TaskManager.Task::(_updateState) Task=`a7f71b8b-f14f-40d2-9d29-7bf308cda21c`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:17,305::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:17,305::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:17,306::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:17,306::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:17,307::task::1185::TaskManager.Task::(prepare) Task=`a7f71b8b-f14f-40d2-9d29-7bf308cda21c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:17,307::task::595::TaskManager.Task::(_updateState) Task=`a7f71b8b-f14f-40d2-9d29-7bf308cda21c`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:17,307::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:17,308::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:17,308::task::990::TaskManager.Task::(_decref) Task=`a7f71b8b-f14f-40d2-9d29-7bf308cda21c`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:18,352::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:18,354::task::595::TaskManager.Task::(_updateState) Task=`d38824b0-132b-447b-8201-8c6482c58225`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:18,354::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:18,354::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:18,355::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:18,355::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:18,355::task::1185::TaskManager.Task::(prepare) Task=`d38824b0-132b-447b-8201-8c6482c58225`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:18,356::task::595::TaskManager.Task::(_updateState) Task=`d38824b0-132b-447b-8201-8c6482c58225`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:18,356::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:18,356::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:18,356::task::990::TaskManager.Task::(_decref) Task=`d38824b0-132b-447b-8201-8c6482c58225`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:19,360::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:19,361::task::595::TaskManager.Task::(_updateState) Task=`25510db7-a778-4d84-8bbd-c9b2da6157f8`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:19,361::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:19,361::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:19,362::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:19,362::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:19,362::task::1185::TaskManager.Task::(prepare) Task=`25510db7-a778-4d84-8bbd-c9b2da6157f8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:19,363::task::595::TaskManager.Task::(_updateState) Task=`25510db7-a778-4d84-8bbd-c9b2da6157f8`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:19,363::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:19,363::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:19,364::task::990::TaskManager.Task::(_decref) Task=`25510db7-a778-4d84-8bbd-c9b2da6157f8`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:20,319::task::595::TaskManager.Task::(_updateState) Task=`9407ce89-7b11-4fa4-bd20-62644f2b46f0`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:20,319::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-13::INFO::2014-04-11 22:01:20,320::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '49.1', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000337815', 'lastCheck': '8.4', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '10.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.7', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '27.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000239907', 'lastCheck': '6.9', 'valid': True}}
>Thread-13::DEBUG::2014-04-11 22:01:20,320::task::1185::TaskManager.Task::(prepare) Task=`9407ce89-7b11-4fa4-bd20-62644f2b46f0`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '49.1', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000337815', 'lastCheck': '8.4', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '10.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.7', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '27.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000239907', 'lastCheck': '6.9', 'valid': True}}
>Thread-13::DEBUG::2014-04-11 22:01:20,321::task::595::TaskManager.Task::(_updateState) Task=`9407ce89-7b11-4fa4-bd20-62644f2b46f0`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:20,321::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:20,321::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:20,322::task::990::TaskManager.Task::(_decref) Task=`9407ce89-7b11-4fa4-bd20-62644f2b46f0`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:20,371::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:20,372::task::595::TaskManager.Task::(_updateState) Task=`fdd6de9d-916f-4d5b-9d6a-cc5bcc05dc4c`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:20,372::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:20,372::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:20,373::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:20,373::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:20,373::task::1185::TaskManager.Task::(prepare) Task=`fdd6de9d-916f-4d5b-9d6a-cc5bcc05dc4c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:20,373::task::595::TaskManager.Task::(_updateState) Task=`fdd6de9d-916f-4d5b-9d6a-cc5bcc05dc4c`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:20,374::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:20,374::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:20,374::task::990::TaskManager.Task::(_decref) Task=`fdd6de9d-916f-4d5b-9d6a-cc5bcc05dc4c`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:21,418::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:21,419::task::595::TaskManager.Task::(_updateState) Task=`1098356d-057a-40e2-9ab7-a2403466a51e`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:21,419::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:21,420::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:21,420::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:21,420::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:21,421::task::1185::TaskManager.Task::(prepare) Task=`1098356d-057a-40e2-9ab7-a2403466a51e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:21,421::task::595::TaskManager.Task::(_updateState) Task=`1098356d-057a-40e2-9ab7-a2403466a51e`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:21,421::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:21,422::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:21,422::task::990::TaskManager.Task::(_decref) Task=`1098356d-057a-40e2-9ab7-a2403466a51e`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:01:21,985::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:01:21,999::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000277253 s, 1.3 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:22,427::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:22,428::task::595::TaskManager.Task::(_updateState) Task=`0ff766f9-56cc-49c1-85d7-8bc9572ada49`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:22,429::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:22,429::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:22,430::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:22,430::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:22,430::task::1185::TaskManager.Task::(prepare) Task=`0ff766f9-56cc-49c1-85d7-8bc9572ada49`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:22,431::task::595::TaskManager.Task::(_updateState) Task=`0ff766f9-56cc-49c1-85d7-8bc9572ada49`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:22,431::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:22,432::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:22,432::task::990::TaskManager.Task::(_decref) Task=`0ff766f9-56cc-49c1-85d7-8bc9572ada49`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:23,436::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:23,453::task::595::TaskManager.Task::(_updateState) Task=`c6548bf0-5af2-4cf7-a657-6c6ce3ddae34`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:23,454::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:23,456::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:23,460::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:23,464::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:23,468::task::1185::TaskManager.Task::(prepare) Task=`c6548bf0-5af2-4cf7-a657-6c6ce3ddae34`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:23,472::task::595::TaskManager.Task::(_updateState) Task=`c6548bf0-5af2-4cf7-a657-6c6ce3ddae34`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:23,476::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:23,479::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:23,481::task::990::TaskManager.Task::(_decref) Task=`c6548bf0-5af2-4cf7-a657-6c6ce3ddae34`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:01:23,516::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:01:23,541::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000250886 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:24,494::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:24,495::task::595::TaskManager.Task::(_updateState) Task=`8e8f7582-00ac-4a11-914a-6740ed11b0d8`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:24,495::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:24,495::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:24,496::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:24,496::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:24,496::task::1185::TaskManager.Task::(prepare) Task=`8e8f7582-00ac-4a11-914a-6740ed11b0d8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:24,497::task::595::TaskManager.Task::(_updateState) Task=`8e8f7582-00ac-4a11-914a-6740ed11b0d8`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:24,497::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:24,497::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:24,498::task::990::TaskManager.Task::(_decref) Task=`8e8f7582-00ac-4a11-914a-6740ed11b0d8`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:25,502::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:25,502::task::595::TaskManager.Task::(_updateState) Task=`e7bdb12a-4f46-4b80-905e-7d0f43ff74b0`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:25,503::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:25,503::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:25,503::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:25,504::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:25,504::task::1185::TaskManager.Task::(prepare) Task=`e7bdb12a-4f46-4b80-905e-7d0f43ff74b0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:25,504::task::595::TaskManager.Task::(_updateState) Task=`e7bdb12a-4f46-4b80-905e-7d0f43ff74b0`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:25,504::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:25,505::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:25,505::task::990::TaskManager.Task::(_decref) Task=`e7bdb12a-4f46-4b80-905e-7d0f43ff74b0`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:26,511::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:26,513::task::595::TaskManager.Task::(_updateState) Task=`b25f7ec8-75bb-4bdf-ba06-37d9381d55e9`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:26,513::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:26,513::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:26,514::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:26,514::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:26,515::task::1185::TaskManager.Task::(prepare) Task=`b25f7ec8-75bb-4bdf-ba06-37d9381d55e9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:26,515::task::595::TaskManager.Task::(_updateState) Task=`b25f7ec8-75bb-4bdf-ba06-37d9381d55e9`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:26,515::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:26,516::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:26,516::task::990::TaskManager.Task::(_decref) Task=`b25f7ec8-75bb-4bdf-ba06-37d9381d55e9`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:27,521::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:27,522::task::595::TaskManager.Task::(_updateState) Task=`17592de4-b554-462c-8972-d7b7090934e4`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:27,522::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:27,522::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:27,523::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:27,523::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:27,523::task::1185::TaskManager.Task::(prepare) Task=`17592de4-b554-462c-8972-d7b7090934e4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:27,524::task::595::TaskManager.Task::(_updateState) Task=`17592de4-b554-462c-8972-d7b7090934e4`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:27,524::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:27,525::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:27,525::task::990::TaskManager.Task::(_decref) Task=`17592de4-b554-462c-8972-d7b7090934e4`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:28,529::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:28,530::task::595::TaskManager.Task::(_updateState) Task=`4c02caf8-14ee-4659-a8b2-8ede5bf42487`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:28,530::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:28,530::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:28,531::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:28,531::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:28,531::task::1185::TaskManager.Task::(prepare) Task=`4c02caf8-14ee-4659-a8b2-8ede5bf42487`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:28,531::task::595::TaskManager.Task::(_updateState) Task=`4c02caf8-14ee-4659-a8b2-8ede5bf42487`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:28,532::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:28,532::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:28,532::task::990::TaskManager.Task::(_decref) Task=`4c02caf8-14ee-4659-a8b2-8ede5bf42487`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:29,537::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:29,538::task::595::TaskManager.Task::(_updateState) Task=`6345a132-f550-4d4b-8c42-bc2c44ccb23c`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:29,538::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:29,538::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:29,539::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:29,539::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:29,540::task::1185::TaskManager.Task::(prepare) Task=`6345a132-f550-4d4b-8c42-bc2c44ccb23c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:29,540::task::595::TaskManager.Task::(_updateState) Task=`6345a132-f550-4d4b-8c42-bc2c44ccb23c`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:29,541::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:29,541::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:29,541::task::990::TaskManager.Task::(_decref) Task=`6345a132-f550-4d4b-8c42-bc2c44ccb23c`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:30,545::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:30,546::task::595::TaskManager.Task::(_updateState) Task=`da13f63b-45a6-4db1-9e68-68f2a0d96cb4`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:30,546::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:30,547::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:30,547::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:30,547::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:30,548::task::1185::TaskManager.Task::(prepare) Task=`da13f63b-45a6-4db1-9e68-68f2a0d96cb4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:30,548::task::595::TaskManager.Task::(_updateState) Task=`da13f63b-45a6-4db1-9e68-68f2a0d96cb4`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:30,549::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:30,549::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:30,549::task::990::TaskManager.Task::(_decref) Task=`da13f63b-45a6-4db1-9e68-68f2a0d96cb4`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:31,555::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:31,556::task::595::TaskManager.Task::(_updateState) Task=`8afc502f-a6c0-4cb2-b90b-070e80434c32`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:31,557::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:31,557::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:31,558::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:31,559::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:31,559::task::1185::TaskManager.Task::(prepare) Task=`8afc502f-a6c0-4cb2-b90b-070e80434c32`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:31,560::task::595::TaskManager.Task::(_updateState) Task=`8afc502f-a6c0-4cb2-b90b-070e80434c32`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:31,561::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:31,561::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:31,561::task::990::TaskManager.Task::(_decref) Task=`8afc502f-a6c0-4cb2-b90b-070e80434c32`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:01:32,038::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:01:32,054::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000273063 s, 1.4 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:32,567::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:32,570::task::595::TaskManager.Task::(_updateState) Task=`23ee456d-fd6b-48c2-9e03-932941e20344`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:32,571::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:32,571::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:32,572::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:32,572::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:32,572::task::1185::TaskManager.Task::(prepare) Task=`23ee456d-fd6b-48c2-9e03-932941e20344`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:32,573::task::595::TaskManager.Task::(_updateState) Task=`23ee456d-fd6b-48c2-9e03-932941e20344`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:32,573::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:32,574::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:32,575::task::990::TaskManager.Task::(_decref) Task=`23ee456d-fd6b-48c2-9e03-932941e20344`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:01:33,572::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-13::DEBUG::2014-04-11 22:01:33,590::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:33,597::task::595::TaskManager.Task::(_updateState) Task=`6f5d3752-608a-40c2-a826-bd93de918ae1`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:33,602::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:33,606::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:33,610::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:33,613::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:33,617::task::1185::TaskManager.Task::(prepare) Task=`6f5d3752-608a-40c2-a826-bd93de918ae1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:33,621::task::595::TaskManager.Task::(_updateState) Task=`6f5d3752-608a-40c2-a826-bd93de918ae1`::moving from state preparing -> state finished
>Thread-44::DEBUG::2014-04-11 22:01:33,624::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000250837 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:33,625::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:33,630::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:33,632::task::990::TaskManager.Task::(_decref) Task=`6f5d3752-608a-40c2-a826-bd93de918ae1`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:34,647::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:34,649::task::595::TaskManager.Task::(_updateState) Task=`d538c307-bdcc-4d36-8f17-7c0d42120d3c`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:34,650::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:34,651::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:34,651::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:34,652::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:34,652::task::1185::TaskManager.Task::(prepare) Task=`d538c307-bdcc-4d36-8f17-7c0d42120d3c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:34,653::task::595::TaskManager.Task::(_updateState) Task=`d538c307-bdcc-4d36-8f17-7c0d42120d3c`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:34,654::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:34,654::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:34,655::task::990::TaskManager.Task::(_decref) Task=`d538c307-bdcc-4d36-8f17-7c0d42120d3c`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:35,662::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:35,665::task::595::TaskManager.Task::(_updateState) Task=`8bf581da-50e3-4be2-a580-57f6db88e596`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:35,666::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:35,667::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:35,667::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:35,669::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:35,671::task::1185::TaskManager.Task::(prepare) Task=`8bf581da-50e3-4be2-a580-57f6db88e596`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:35,671::task::595::TaskManager.Task::(_updateState) Task=`8bf581da-50e3-4be2-a580-57f6db88e596`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:35,671::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:35,673::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:35,673::task::990::TaskManager.Task::(_decref) Task=`8bf581da-50e3-4be2-a580-57f6db88e596`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:35,765::task::595::TaskManager.Task::(_updateState) Task=`3e15d61c-984d-44c5-90f3-7ebe7553d3dc`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:35,767::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-13::INFO::2014-04-11 22:01:35,769::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '64.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000273063', 'lastCheck': '3.7', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '25.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '42.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000250837', 'lastCheck': '2.1', 'valid': True}}
>Thread-13::DEBUG::2014-04-11 22:01:35,769::task::1185::TaskManager.Task::(prepare) Task=`3e15d61c-984d-44c5-90f3-7ebe7553d3dc`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '64.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000273063', 'lastCheck': '3.7', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '25.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '42.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000250837', 'lastCheck': '2.1', 'valid': True}}
>Thread-13::DEBUG::2014-04-11 22:01:35,771::task::595::TaskManager.Task::(_updateState) Task=`3e15d61c-984d-44c5-90f3-7ebe7553d3dc`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:35,771::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:35,772::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:35,773::task::990::TaskManager.Task::(_decref) Task=`3e15d61c-984d-44c5-90f3-7ebe7553d3dc`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:36,684::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:36,691::task::595::TaskManager.Task::(_updateState) Task=`e44f47d0-1f92-4fe2-899a-af8271c48949`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:36,696::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:36,700::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:36,703::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:36,708::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:36,709::task::1185::TaskManager.Task::(prepare) Task=`e44f47d0-1f92-4fe2-899a-af8271c48949`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:36,711::task::595::TaskManager.Task::(_updateState) Task=`e44f47d0-1f92-4fe2-899a-af8271c48949`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:36,719::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:36,721::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:36,726::task::990::TaskManager.Task::(_decref) Task=`e44f47d0-1f92-4fe2-899a-af8271c48949`::ref 0 aborting False
>Thread-968::WARNING::2014-04-11 22:01:37,001::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-974::DEBUG::2014-04-11 22:01:37,125::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-974::DEBUG::2014-04-11 22:01:37,146::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-974::DEBUG::2014-04-11 22:01:37,155::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-974::DEBUG::2014-04-11 22:01:37,162::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-974::DEBUG::2014-04-11 22:01:37,175::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-974::DEBUG::2014-04-11 22:01:37,186::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-974::DEBUG::2014-04-11 22:01:37,191::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-974::DEBUG::2014-04-11 22:01:37,197::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-974::DEBUG::2014-04-11 22:01:37,235::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-981::WARNING::2014-04-11 22:01:37,379::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-977::ERROR::2014-04-11 22:01:37,384::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-31::DEBUG::2014-04-11 22:01:37,387::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-31::DEBUG::2014-04-11 22:01:37,388::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-31::DEBUG::2014-04-11 22:01:37,389::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-31::DEBUG::2014-04-11 22:01:37,391::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-31::DEBUG::2014-04-11 22:01:37,391::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-31::DEBUG::2014-04-11 22:01:37,392::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-31::DEBUG::2014-04-11 22:01:37,442::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-31::DEBUG::2014-04-11 22:01:37,448::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-988::WARNING::2014-04-11 22:01:37,463::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-988::DEBUG::2014-04-11 22:01:37,468::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-978::ERROR::2014-04-11 22:01:37,470::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-38::DEBUG::2014-04-11 22:01:37,472::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-31::DEBUG::2014-04-11 22:01:37,603::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-31::DEBUG::2014-04-11 22:01:37,604::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-31::DEBUG::2014-04-11 22:01:37,605::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-31::DEBUG::2014-04-11 22:01:37,606::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-31::DEBUG::2014-04-11 22:01:37,606::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-31::DEBUG::2014-04-11 22:01:37,606::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-31::DEBUG::2014-04-11 22:01:37,607::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-31::DEBUG::2014-04-11 22:01:37,608::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-38::DEBUG::2014-04-11 22:01:37,608::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-31::ERROR::2014-04-11 22:01:37,610::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-38::DEBUG::2014-04-11 22:01:37,611::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-31::ERROR::2014-04-11 22:01:37,611::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-38::DEBUG::2014-04-11 22:01:37,614::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-31::DEBUG::2014-04-11 22:01:37,615::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:37,617::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-38::DEBUG::2014-04-11 22:01:37,622::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-31::DEBUG::2014-04-11 22:01:37,637::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-38::DEBUG::2014-04-11 22:01:37,650::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-38::DEBUG::2014-04-11 22:01:37,651::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-38::DEBUG::2014-04-11 22:01:37,710::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-38::DEBUG::2014-04-11 22:01:37,712::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-13::DEBUG::2014-04-11 22:01:37,742::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:37,743::task::595::TaskManager.Task::(_updateState) Task=`b9bd0f68-848f-4986-a6c7-1b607add2b54`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:37,743::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:37,744::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:37,744::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:37,745::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:37,745::task::1185::TaskManager.Task::(prepare) Task=`b9bd0f68-848f-4986-a6c7-1b607add2b54`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:37,745::task::595::TaskManager.Task::(_updateState) Task=`b9bd0f68-848f-4986-a6c7-1b607add2b54`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:37,746::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:37,746::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:37,747::task::990::TaskManager.Task::(_decref) Task=`b9bd0f68-848f-4986-a6c7-1b607add2b54`::ref 0 aborting False
>Thread-31::DEBUG::2014-04-11 22:01:37,944::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-31::WARNING::2014-04-11 22:01:37,946::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-31::DEBUG::2014-04-11 22:01:37,947::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:37,951::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:37,959::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:37,989::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:38,011::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:38,021::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:38,022::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:38,022::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-38::ERROR::2014-04-11 22:01:38,023::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-38::ERROR::2014-04-11 22:01:38,023::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-38::DEBUG::2014-04-11 22:01:38,024::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-38::DEBUG::2014-04-11 22:01:38,025::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-38::DEBUG::2014-04-11 22:01:38,306::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-38::WARNING::2014-04-11 22:01:38,308::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-38::DEBUG::2014-04-11 22:01:38,309::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-13::DEBUG::2014-04-11 22:01:38,753::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:38,754::task::595::TaskManager.Task::(_updateState) Task=`32e33d54-c63a-4281-aa05-6bda21433af4`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:38,755::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:38,755::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:38,756::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:38,757::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:38,757::task::1185::TaskManager.Task::(prepare) Task=`32e33d54-c63a-4281-aa05-6bda21433af4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:38,757::task::595::TaskManager.Task::(_updateState) Task=`32e33d54-c63a-4281-aa05-6bda21433af4`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:38,758::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:38,758::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:38,759::task::990::TaskManager.Task::(_decref) Task=`32e33d54-c63a-4281-aa05-6bda21433af4`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:39,764::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:39,765::task::595::TaskManager.Task::(_updateState) Task=`3fe5aae2-325b-4f68-bd80-abcd43752457`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:39,765::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:39,766::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:39,766::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:39,767::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:39,767::task::1185::TaskManager.Task::(prepare) Task=`3fe5aae2-325b-4f68-bd80-abcd43752457`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:39,768::task::595::TaskManager.Task::(_updateState) Task=`3fe5aae2-325b-4f68-bd80-abcd43752457`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:39,768::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:39,768::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:39,769::task::990::TaskManager.Task::(_decref) Task=`3fe5aae2-325b-4f68-bd80-abcd43752457`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:40,774::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:40,775::task::595::TaskManager.Task::(_updateState) Task=`0d3e91b9-f0a1-4d1d-a52d-a96678bceacf`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:40,776::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:40,777::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:40,777::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:40,778::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:40,779::task::1185::TaskManager.Task::(prepare) Task=`0d3e91b9-f0a1-4d1d-a52d-a96678bceacf`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:40,779::task::595::TaskManager.Task::(_updateState) Task=`0d3e91b9-f0a1-4d1d-a52d-a96678bceacf`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:40,780::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:40,781::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:40,781::task::990::TaskManager.Task::(_decref) Task=`0d3e91b9-f0a1-4d1d-a52d-a96678bceacf`::ref 0 aborting False
>Thread-20::ERROR::2014-04-11 22:01:41,241::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-20::DEBUG::2014-04-11 22:01:41,242::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 changed its status to Invalid
>Thread-1008::DEBUG::2014-04-11 22:01:41,243::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-20::DEBUG::2014-04-11 22:01:41,244::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-1008::DEBUG::2014-04-11 22:01:41,257::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-1008::DEBUG::2014-04-11 22:01:41,259::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_upgradePoolDomain`
>Thread-1008::DEBUG::2014-04-11 22:01:41,261::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-13::DEBUG::2014-04-11 22:01:41,788::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:41,789::task::595::TaskManager.Task::(_updateState) Task=`1267a86e-0d04-4ace-9081-8abe16bc6833`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:41,790::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:41,790::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:41,790::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:41,791::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:41,791::task::1185::TaskManager.Task::(prepare) Task=`1267a86e-0d04-4ace-9081-8abe16bc6833`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:41,792::task::595::TaskManager.Task::(_updateState) Task=`1267a86e-0d04-4ace-9081-8abe16bc6833`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:41,792::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:41,792::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:41,793::task::990::TaskManager.Task::(_decref) Task=`1267a86e-0d04-4ace-9081-8abe16bc6833`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:01:42,085::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:01:42,104::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000265116 s, 1.4 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:42,798::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:42,814::task::595::TaskManager.Task::(_updateState) Task=`a9543da0-ec8e-4ba9-b6c7-a06f0bddd202`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:42,815::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:42,815::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:42,816::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:42,816::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:42,816::task::1185::TaskManager.Task::(prepare) Task=`a9543da0-ec8e-4ba9-b6c7-a06f0bddd202`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:42,817::task::595::TaskManager.Task::(_updateState) Task=`a9543da0-ec8e-4ba9-b6c7-a06f0bddd202`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:42,818::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:42,818::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:42,818::task::990::TaskManager.Task::(_decref) Task=`a9543da0-ec8e-4ba9-b6c7-a06f0bddd202`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:01:43,670::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:01:43,683::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000273416 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:01:43,823::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:43,824::task::595::TaskManager.Task::(_updateState) Task=`2493e9cb-7ab6-4f77-b13b-0e75fc1e23c6`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:43,826::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:43,826::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:43,826::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:43,827::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:43,827::task::1185::TaskManager.Task::(prepare) Task=`2493e9cb-7ab6-4f77-b13b-0e75fc1e23c6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:43,827::task::595::TaskManager.Task::(_updateState) Task=`2493e9cb-7ab6-4f77-b13b-0e75fc1e23c6`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:43,828::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:43,828::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:43,828::task::990::TaskManager.Task::(_decref) Task=`2493e9cb-7ab6-4f77-b13b-0e75fc1e23c6`::ref 0 aborting False
>Thread-13::DEBUG::2014-04-11 22:01:44,833::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-13::DEBUG::2014-04-11 22:01:44,834::task::595::TaskManager.Task::(_updateState) Task=`9c65ec9b-75f4-427d-9581-d8d7e200a487`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:01:44,834::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-13::DEBUG::2014-04-11 22:01:44,834::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-13::DEBUG::2014-04-11 22:01:44,835::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-13::INFO::2014-04-11 22:01:44,835::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:44,835::task::1185::TaskManager.Task::(prepare) Task=`9c65ec9b-75f4-427d-9581-d8d7e200a487`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-13::DEBUG::2014-04-11 22:01:44,836::task::595::TaskManager.Task::(_updateState) Task=`9c65ec9b-75f4-427d-9581-d8d7e200a487`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:01:44,836::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:01:44,836::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:01:44,837::task::990::TaskManager.Task::(_decref) Task=`9c65ec9b-75f4-427d-9581-d8d7e200a487`::ref 0 aborting False
>Thread-155::ERROR::2014-04-11 22:01:44,987::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-155::ERROR::2014-04-11 22:01:44,990::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-155::DEBUG::2014-04-11 22:01:44,990::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain}
>Thread-155::INFO::2014-04-11 22:01:44,991::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}]}
>Thread-155::DEBUG::2014-04-11 22:01:44,991::task::1185::TaskManager.Task::(prepare) Task=`49488c60-1e2d-446c-b2c8-d8ce81440468`::finished: {'statuslist': [{'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}]}
>Thread-155::DEBUG::2014-04-11 22:01:44,992::task::595::TaskManager.Task::(_updateState) Task=`49488c60-1e2d-446c-b2c8-d8ce81440468`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:44,992::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:44,992::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:44,993::task::990::TaskManager.Task::(_decref) Task=`49488c60-1e2d-446c-b2c8-d8ce81440468`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:45,841::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:45,842::task::595::TaskManager.Task::(_updateState) Task=`bdce2879-b610-4150-bdba-eed36dd91f3e`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:45,842::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:45,843::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:45,843::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:45,843::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:45,844::task::1185::TaskManager.Task::(prepare) Task=`bdce2879-b610-4150-bdba-eed36dd91f3e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:45,844::task::595::TaskManager.Task::(_updateState) Task=`bdce2879-b610-4150-bdba-eed36dd91f3e`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:45,844::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:45,845::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:45,845::task::990::TaskManager.Task::(_decref) Task=`bdce2879-b610-4150-bdba-eed36dd91f3e`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:46,856::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:46,857::task::595::TaskManager.Task::(_updateState) Task=`f69a27b2-a38e-408e-a51d-c0c22f079da7`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:46,857::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:46,858::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:46,858::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:46,858::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:46,859::task::1185::TaskManager.Task::(prepare) Task=`f69a27b2-a38e-408e-a51d-c0c22f079da7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:46,859::task::595::TaskManager.Task::(_updateState) Task=`f69a27b2-a38e-408e-a51d-c0c22f079da7`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:46,859::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:46,860::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:46,860::task::990::TaskManager.Task::(_decref) Task=`f69a27b2-a38e-408e-a51d-c0c22f079da7`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:47,865::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:47,866::task::595::TaskManager.Task::(_updateState) Task=`a80c4a15-a7ff-4a9e-9ddd-476e07f136a8`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:47,866::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:47,866::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:47,867::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:47,867::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:47,867::task::1185::TaskManager.Task::(prepare) Task=`a80c4a15-a7ff-4a9e-9ddd-476e07f136a8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:47,867::task::595::TaskManager.Task::(_updateState) Task=`a80c4a15-a7ff-4a9e-9ddd-476e07f136a8`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:47,868::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:47,868::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:47,868::task::990::TaskManager.Task::(_decref) Task=`a80c4a15-a7ff-4a9e-9ddd-476e07f136a8`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:48,904::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:48,906::task::595::TaskManager.Task::(_updateState) Task=`540da1e6-f026-4bc1-8a03-e2ebc5343e8c`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:48,907::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:48,907::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:48,907::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:48,908::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:48,908::task::1185::TaskManager.Task::(prepare) Task=`540da1e6-f026-4bc1-8a03-e2ebc5343e8c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:48,908::task::595::TaskManager.Task::(_updateState) Task=`540da1e6-f026-4bc1-8a03-e2ebc5343e8c`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:48,908::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:48,909::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:48,909::task::990::TaskManager.Task::(_decref) Task=`540da1e6-f026-4bc1-8a03-e2ebc5343e8c`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:49,914::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:49,915::task::595::TaskManager.Task::(_updateState) Task=`36062093-f7ea-4107-be4d-1904abc4628a`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:49,915::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:49,915::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:49,916::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:49,916::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:49,916::task::1185::TaskManager.Task::(prepare) Task=`36062093-f7ea-4107-be4d-1904abc4628a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:49,917::task::595::TaskManager.Task::(_updateState) Task=`36062093-f7ea-4107-be4d-1904abc4628a`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:49,917::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:49,917::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:49,917::task::990::TaskManager.Task::(_decref) Task=`36062093-f7ea-4107-be4d-1904abc4628a`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:50,922::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:50,923::task::595::TaskManager.Task::(_updateState) Task=`bd63bc78-e5cc-4151-b220-5ab7f462c7b0`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:50,923::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:50,923::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:50,923::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:50,924::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:50,924::task::1185::TaskManager.Task::(prepare) Task=`bd63bc78-e5cc-4151-b220-5ab7f462c7b0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:50,924::task::595::TaskManager.Task::(_updateState) Task=`bd63bc78-e5cc-4151-b220-5ab7f462c7b0`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:50,925::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:50,925::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:50,925::task::990::TaskManager.Task::(_decref) Task=`bd63bc78-e5cc-4151-b220-5ab7f462c7b0`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:51,182::task::595::TaskManager.Task::(_updateState) Task=`e31f6fc3-d293-40b6-ab92-ccb8b32b385a`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:51,182::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-155::INFO::2014-04-11 22:01:51,183::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.9', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000265116', 'lastCheck': '9.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '79.5', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '41.4', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '79.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '58.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000273416', 'lastCheck': '7.5', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:01:51,183::task::1185::TaskManager.Task::(prepare) Task=`e31f6fc3-d293-40b6-ab92-ccb8b32b385a`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.9', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000265116', 'lastCheck': '9.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '79.5', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '41.4', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '79.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '58.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000273416', 'lastCheck': '7.5', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:01:51,184::task::595::TaskManager.Task::(_updateState) Task=`e31f6fc3-d293-40b6-ab92-ccb8b32b385a`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:51,184::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:51,184::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:51,185::task::990::TaskManager.Task::(_decref) Task=`e31f6fc3-d293-40b6-ab92-ccb8b32b385a`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:51,930::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:51,932::task::595::TaskManager.Task::(_updateState) Task=`8a4464d2-fb78-45c4-becb-2b9af2341346`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:51,932::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:51,933::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:51,933::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:51,934::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:51,934::task::1185::TaskManager.Task::(prepare) Task=`8a4464d2-fb78-45c4-becb-2b9af2341346`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:51,935::task::595::TaskManager.Task::(_updateState) Task=`8a4464d2-fb78-45c4-becb-2b9af2341346`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:51,935::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:51,935::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:51,936::task::990::TaskManager.Task::(_decref) Task=`8a4464d2-fb78-45c4-becb-2b9af2341346`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:01:52,137::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:01:52,152::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000293023 s, 1.3 MB/s\n'; <rc> = 0
>Thread-155::DEBUG::2014-04-11 22:01:52,941::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:52,942::task::595::TaskManager.Task::(_updateState) Task=`22c17f8c-b846-49fc-a1b6-8fe3fa940161`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:52,943::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:52,943::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:52,944::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:52,944::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:52,944::task::1185::TaskManager.Task::(prepare) Task=`22c17f8c-b846-49fc-a1b6-8fe3fa940161`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:52,944::task::595::TaskManager.Task::(_updateState) Task=`22c17f8c-b846-49fc-a1b6-8fe3fa940161`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:52,945::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:52,945::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:52,945::task::990::TaskManager.Task::(_decref) Task=`22c17f8c-b846-49fc-a1b6-8fe3fa940161`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:01:53,713::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:01:53,730::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000342916 s, 2.2 MB/s\n'; <rc> = 0
>Thread-155::DEBUG::2014-04-11 22:01:53,950::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:53,951::task::595::TaskManager.Task::(_updateState) Task=`2d48ccc0-a68d-4537-b082-b3a86cda3c24`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:53,952::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:53,952::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:53,953::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:53,953::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:53,953::task::1185::TaskManager.Task::(prepare) Task=`2d48ccc0-a68d-4537-b082-b3a86cda3c24`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:53,954::task::595::TaskManager.Task::(_updateState) Task=`2d48ccc0-a68d-4537-b082-b3a86cda3c24`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:53,955::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:53,955::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:53,955::task::990::TaskManager.Task::(_decref) Task=`2d48ccc0-a68d-4537-b082-b3a86cda3c24`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:54,960::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:54,961::task::595::TaskManager.Task::(_updateState) Task=`3b40b2cb-dff2-4746-9b70-c7064f8fdf59`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:54,961::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:54,961::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:54,962::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:54,962::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:54,962::task::1185::TaskManager.Task::(prepare) Task=`3b40b2cb-dff2-4746-9b70-c7064f8fdf59`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:54,963::task::595::TaskManager.Task::(_updateState) Task=`3b40b2cb-dff2-4746-9b70-c7064f8fdf59`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:54,963::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:54,963::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:54,964::task::990::TaskManager.Task::(_decref) Task=`3b40b2cb-dff2-4746-9b70-c7064f8fdf59`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:55,969::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:55,969::task::595::TaskManager.Task::(_updateState) Task=`804e8e1b-97ac-493d-a1f3-5e53776eb71f`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:55,970::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:55,970::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:55,970::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:55,971::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:55,971::task::1185::TaskManager.Task::(prepare) Task=`804e8e1b-97ac-493d-a1f3-5e53776eb71f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:55,971::task::595::TaskManager.Task::(_updateState) Task=`804e8e1b-97ac-493d-a1f3-5e53776eb71f`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:55,972::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:55,972::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:55,972::task::990::TaskManager.Task::(_decref) Task=`804e8e1b-97ac-493d-a1f3-5e53776eb71f`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:56,977::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:56,979::task::595::TaskManager.Task::(_updateState) Task=`957aa3e4-1587-4adc-88d5-a8cd05be08cd`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:56,979::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:56,980::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:56,980::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:56,981::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:56,981::task::1185::TaskManager.Task::(prepare) Task=`957aa3e4-1587-4adc-88d5-a8cd05be08cd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:56,981::task::595::TaskManager.Task::(_updateState) Task=`957aa3e4-1587-4adc-88d5-a8cd05be08cd`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:56,982::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:56,982::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:56,982::task::990::TaskManager.Task::(_decref) Task=`957aa3e4-1587-4adc-88d5-a8cd05be08cd`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:57,987::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:57,989::task::595::TaskManager.Task::(_updateState) Task=`77016bb1-dc25-4510-86a8-855cf9a1d207`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:57,989::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:57,989::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:57,990::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:57,990::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:57,990::task::1185::TaskManager.Task::(prepare) Task=`77016bb1-dc25-4510-86a8-855cf9a1d207`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:57,991::task::595::TaskManager.Task::(_updateState) Task=`77016bb1-dc25-4510-86a8-855cf9a1d207`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:57,991::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:57,991::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:57,992::task::990::TaskManager.Task::(_decref) Task=`77016bb1-dc25-4510-86a8-855cf9a1d207`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:01:58,996::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:01:58,997::task::595::TaskManager.Task::(_updateState) Task=`fc2782e8-4bd7-4d80-8fc4-cd3ab655ead1`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:01:58,997::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:01:58,998::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:01:58,998::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:01:58,998::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:58,999::task::1185::TaskManager.Task::(prepare) Task=`fc2782e8-4bd7-4d80-8fc4-cd3ab655ead1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:01:58,999::task::595::TaskManager.Task::(_updateState) Task=`fc2782e8-4bd7-4d80-8fc4-cd3ab655ead1`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:01:58,999::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:01:59,000::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:01:59,000::task::990::TaskManager.Task::(_decref) Task=`fc2782e8-4bd7-4d80-8fc4-cd3ab655ead1`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:00,004::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:00,005::task::595::TaskManager.Task::(_updateState) Task=`90aca4f8-9bb3-4b9d-bc48-cde3c189a242`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:00,005::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:00,006::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:00,006::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:00,006::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:00,007::task::1185::TaskManager.Task::(prepare) Task=`90aca4f8-9bb3-4b9d-bc48-cde3c189a242`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:00,007::task::595::TaskManager.Task::(_updateState) Task=`90aca4f8-9bb3-4b9d-bc48-cde3c189a242`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:00,007::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:00,008::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:00,008::task::990::TaskManager.Task::(_decref) Task=`90aca4f8-9bb3-4b9d-bc48-cde3c189a242`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:01,013::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:01,013::task::595::TaskManager.Task::(_updateState) Task=`c6da1f71-1d0d-44d5-9e9e-b51db9c21ff4`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:01,014::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:01,014::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:01,015::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:01,015::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:01,015::task::1185::TaskManager.Task::(prepare) Task=`c6da1f71-1d0d-44d5-9e9e-b51db9c21ff4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:01,016::task::595::TaskManager.Task::(_updateState) Task=`c6da1f71-1d0d-44d5-9e9e-b51db9c21ff4`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:01,016::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:01,016::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:01,017::task::990::TaskManager.Task::(_decref) Task=`c6da1f71-1d0d-44d5-9e9e-b51db9c21ff4`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:02,025::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:02,030::task::595::TaskManager.Task::(_updateState) Task=`715fa805-cb26-4bff-9c08-36f6713c8700`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:02,034::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:02,036::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:02,038::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:02,041::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:02,043::task::1185::TaskManager.Task::(prepare) Task=`715fa805-cb26-4bff-9c08-36f6713c8700`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:02,046::task::595::TaskManager.Task::(_updateState) Task=`715fa805-cb26-4bff-9c08-36f6713c8700`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:02,048::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:02,050::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:02,053::task::990::TaskManager.Task::(_decref) Task=`715fa805-cb26-4bff-9c08-36f6713c8700`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:02:02,357::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:02:02,400::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000276526 s, 1.3 MB/s\n'; <rc> = 0
>Thread-43::ERROR::2014-04-11 22:02:02,979::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain a033c2ac-0d01-490c-9552-99ca53d6a64a monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-43::DEBUG::2014-04-11 22:02:02,981::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain a033c2ac-0d01-490c-9552-99ca53d6a64a changed its status to Invalid
>Thread-1011::DEBUG::2014-04-11 22:02:02,983::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-1011::DEBUG::2014-04-11 22:02:02,986::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-1011::DEBUG::2014-04-11 22:02:02,988::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_upgradePoolDomain`
>Thread-1011::DEBUG::2014-04-11 22:02:02,989::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-155::DEBUG::2014-04-11 22:02:03,065::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:03,066::task::595::TaskManager.Task::(_updateState) Task=`3677bb8f-f130-46ac-a942-7771a4312d1d`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:03,066::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:03,067::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:03,067::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:03,068::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:03,068::task::1185::TaskManager.Task::(prepare) Task=`3677bb8f-f130-46ac-a942-7771a4312d1d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:03,069::task::595::TaskManager.Task::(_updateState) Task=`3677bb8f-f130-46ac-a942-7771a4312d1d`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:03,069::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:03,070::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:03,070::task::990::TaskManager.Task::(_decref) Task=`3677bb8f-f130-46ac-a942-7771a4312d1d`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:02:03,766::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:02:03,781::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000265086 s, 2.8 MB/s\n'; <rc> = 0
>Thread-155::DEBUG::2014-04-11 22:02:04,075::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:04,076::task::595::TaskManager.Task::(_updateState) Task=`908b8349-c627-48c7-9e8b-75cd7cb202ca`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:04,077::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:04,077::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:04,077::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:04,078::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:04,078::task::1185::TaskManager.Task::(prepare) Task=`908b8349-c627-48c7-9e8b-75cd7cb202ca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:04,078::task::595::TaskManager.Task::(_updateState) Task=`908b8349-c627-48c7-9e8b-75cd7cb202ca`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:04,079::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:04,079::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:04,079::task::990::TaskManager.Task::(_decref) Task=`908b8349-c627-48c7-9e8b-75cd7cb202ca`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:05,084::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:05,085::task::595::TaskManager.Task::(_updateState) Task=`441ae0b9-ecc4-432f-92e9-e1c45f501024`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:05,085::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:05,085::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:05,086::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:05,086::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:05,086::task::1185::TaskManager.Task::(prepare) Task=`441ae0b9-ecc4-432f-92e9-e1c45f501024`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:05,087::task::595::TaskManager.Task::(_updateState) Task=`441ae0b9-ecc4-432f-92e9-e1c45f501024`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:05,087::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:05,087::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:05,088::task::990::TaskManager.Task::(_decref) Task=`441ae0b9-ecc4-432f-92e9-e1c45f501024`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:06,092::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:06,093::task::595::TaskManager.Task::(_updateState) Task=`83e96f57-9f41-4329-86f9-44f1fa06c7f7`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:06,093::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:06,094::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:06,094::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:06,094::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:06,094::task::1185::TaskManager.Task::(prepare) Task=`83e96f57-9f41-4329-86f9-44f1fa06c7f7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:06,095::task::595::TaskManager.Task::(_updateState) Task=`83e96f57-9f41-4329-86f9-44f1fa06c7f7`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:06,095::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:06,095::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:06,096::task::990::TaskManager.Task::(_decref) Task=`83e96f57-9f41-4329-86f9-44f1fa06c7f7`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:06,531::task::595::TaskManager.Task::(_updateState) Task=`591d1b40-c01c-47f1-9a0c-a988a168f689`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:06,532::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-155::INFO::2014-04-11 22:02:06,532::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '25.3', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000276526', 'lastCheck': '4.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '94.8', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '56.7', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '94.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.6', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000265086', 'lastCheck': '2.7', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:02:06,533::task::1185::TaskManager.Task::(prepare) Task=`591d1b40-c01c-47f1-9a0c-a988a168f689`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '25.3', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000276526', 'lastCheck': '4.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '94.8', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '56.7', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '94.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.6', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000265086', 'lastCheck': '2.7', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:02:06,533::task::595::TaskManager.Task::(_updateState) Task=`591d1b40-c01c-47f1-9a0c-a988a168f689`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:06,533::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:06,534::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:06,534::task::990::TaskManager.Task::(_decref) Task=`591d1b40-c01c-47f1-9a0c-a988a168f689`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:07,100::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:07,101::task::595::TaskManager.Task::(_updateState) Task=`354ddccf-f915-452e-a434-08cf6e65c08b`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:07,102::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:07,102::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:07,103::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:07,103::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:07,103::task::1185::TaskManager.Task::(prepare) Task=`354ddccf-f915-452e-a434-08cf6e65c08b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:07,104::task::595::TaskManager.Task::(_updateState) Task=`354ddccf-f915-452e-a434-08cf6e65c08b`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:07,104::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:07,105::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:07,105::task::990::TaskManager.Task::(_decref) Task=`354ddccf-f915-452e-a434-08cf6e65c08b`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:08,109::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:08,110::task::595::TaskManager.Task::(_updateState) Task=`e67c32db-67d3-43df-b177-05338f8c05b4`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:08,110::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:08,111::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:08,111::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:08,111::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:08,112::task::1185::TaskManager.Task::(prepare) Task=`e67c32db-67d3-43df-b177-05338f8c05b4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:08,112::task::595::TaskManager.Task::(_updateState) Task=`e67c32db-67d3-43df-b177-05338f8c05b4`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:08,112::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:08,113::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:08,113::task::990::TaskManager.Task::(_decref) Task=`e67c32db-67d3-43df-b177-05338f8c05b4`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:09,117::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:09,118::task::595::TaskManager.Task::(_updateState) Task=`ce387f57-8f67-4e2b-9e6d-3a6c1b2e61ab`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:09,118::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:09,118::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:09,119::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:09,119::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:09,119::task::1185::TaskManager.Task::(prepare) Task=`ce387f57-8f67-4e2b-9e6d-3a6c1b2e61ab`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:09,120::task::595::TaskManager.Task::(_updateState) Task=`ce387f57-8f67-4e2b-9e6d-3a6c1b2e61ab`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:09,120::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:09,120::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:09,121::task::990::TaskManager.Task::(_decref) Task=`ce387f57-8f67-4e2b-9e6d-3a6c1b2e61ab`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:10,124::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:10,125::task::595::TaskManager.Task::(_updateState) Task=`536a39ae-18ed-4ede-8ef5-e25e9b5f1f34`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:10,125::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:10,126::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:10,126::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:10,126::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:10,127::task::1185::TaskManager.Task::(prepare) Task=`536a39ae-18ed-4ede-8ef5-e25e9b5f1f34`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:10,127::task::595::TaskManager.Task::(_updateState) Task=`536a39ae-18ed-4ede-8ef5-e25e9b5f1f34`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:10,127::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:10,127::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:10,128::task::990::TaskManager.Task::(_decref) Task=`536a39ae-18ed-4ede-8ef5-e25e9b5f1f34`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:11,132::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:11,133::task::595::TaskManager.Task::(_updateState) Task=`a2dc7b58-a809-4d45-b29d-5d9cf32cca45`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:11,133::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:11,134::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:11,134::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:11,134::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:11,134::task::1185::TaskManager.Task::(prepare) Task=`a2dc7b58-a809-4d45-b29d-5d9cf32cca45`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:11,135::task::595::TaskManager.Task::(_updateState) Task=`a2dc7b58-a809-4d45-b29d-5d9cf32cca45`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:11,135::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:11,135::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:11,136::task::990::TaskManager.Task::(_decref) Task=`a2dc7b58-a809-4d45-b29d-5d9cf32cca45`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:12,141::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:12,142::task::595::TaskManager.Task::(_updateState) Task=`906ac308-0e79-42ef-9893-3c746eb4eca5`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:12,143::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:12,143::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:12,143::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:12,144::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:12,144::task::1185::TaskManager.Task::(prepare) Task=`906ac308-0e79-42ef-9893-3c746eb4eca5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:12,145::task::595::TaskManager.Task::(_updateState) Task=`906ac308-0e79-42ef-9893-3c746eb4eca5`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:12,145::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:12,145::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:12,146::task::990::TaskManager.Task::(_decref) Task=`906ac308-0e79-42ef-9893-3c746eb4eca5`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:02:12,442::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:02:12,457::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000302115 s, 1.2 MB/s\n'; <rc> = 0
>Thread-155::DEBUG::2014-04-11 22:02:13,151::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:13,153::task::595::TaskManager.Task::(_updateState) Task=`4cb856a7-aed7-4666-bf8d-848227c5cd12`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:13,154::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:13,154::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:13,155::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:13,155::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:13,155::task::1185::TaskManager.Task::(prepare) Task=`4cb856a7-aed7-4666-bf8d-848227c5cd12`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:13,156::task::595::TaskManager.Task::(_updateState) Task=`4cb856a7-aed7-4666-bf8d-848227c5cd12`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:13,156::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:13,156::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:13,158::task::990::TaskManager.Task::(_decref) Task=`4cb856a7-aed7-4666-bf8d-848227c5cd12`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:02:13,810::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:02:13,825::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000279867 s, 2.7 MB/s\n'; <rc> = 0
>Thread-155::DEBUG::2014-04-11 22:02:14,163::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:14,165::task::595::TaskManager.Task::(_updateState) Task=`4e4714b7-847a-4c22-952e-55db0958b628`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:14,166::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:14,166::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:14,166::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:14,167::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:14,168::task::1185::TaskManager.Task::(prepare) Task=`4e4714b7-847a-4c22-952e-55db0958b628`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:14,168::task::595::TaskManager.Task::(_updateState) Task=`4e4714b7-847a-4c22-952e-55db0958b628`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:14,168::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:14,169::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:14,169::task::990::TaskManager.Task::(_decref) Task=`4e4714b7-847a-4c22-952e-55db0958b628`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:15,174::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:15,175::task::595::TaskManager.Task::(_updateState) Task=`e89b9630-ac3d-4213-9203-197915d7d755`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:15,175::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:15,176::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:15,176::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:15,176::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:15,177::task::1185::TaskManager.Task::(prepare) Task=`e89b9630-ac3d-4213-9203-197915d7d755`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:15,177::task::595::TaskManager.Task::(_updateState) Task=`e89b9630-ac3d-4213-9203-197915d7d755`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:15,177::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:15,178::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:15,178::task::990::TaskManager.Task::(_decref) Task=`e89b9630-ac3d-4213-9203-197915d7d755`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:16,183::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:16,184::task::595::TaskManager.Task::(_updateState) Task=`a6ac205f-e587-40d0-9bbd-d8ce41032c7b`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:16,184::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:16,185::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:16,185::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:16,185::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:16,186::task::1185::TaskManager.Task::(prepare) Task=`a6ac205f-e587-40d0-9bbd-d8ce41032c7b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:16,186::task::595::TaskManager.Task::(_updateState) Task=`a6ac205f-e587-40d0-9bbd-d8ce41032c7b`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:16,186::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:16,187::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:16,187::task::990::TaskManager.Task::(_decref) Task=`a6ac205f-e587-40d0-9bbd-d8ce41032c7b`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:17,192::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:17,194::task::595::TaskManager.Task::(_updateState) Task=`7a64a7b0-dcf8-42cc-8bf4-8fee283002f5`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:17,194::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:17,195::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:17,196::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:17,197::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:17,197::task::1185::TaskManager.Task::(prepare) Task=`7a64a7b0-dcf8-42cc-8bf4-8fee283002f5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:17,197::task::595::TaskManager.Task::(_updateState) Task=`7a64a7b0-dcf8-42cc-8bf4-8fee283002f5`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:17,198::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:17,198::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:17,199::task::990::TaskManager.Task::(_decref) Task=`7a64a7b0-dcf8-42cc-8bf4-8fee283002f5`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:18,204::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:18,205::task::595::TaskManager.Task::(_updateState) Task=`750c5398-819d-4eac-85d9-341efddbbef9`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:18,205::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:18,206::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:18,206::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:18,207::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:18,207::task::1185::TaskManager.Task::(prepare) Task=`750c5398-819d-4eac-85d9-341efddbbef9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:18,208::task::595::TaskManager.Task::(_updateState) Task=`750c5398-819d-4eac-85d9-341efddbbef9`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:18,208::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:18,209::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:18,209::task::990::TaskManager.Task::(_decref) Task=`750c5398-819d-4eac-85d9-341efddbbef9`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:19,219::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:19,225::task::595::TaskManager.Task::(_updateState) Task=`24a2f822-d893-48b7-9bf6-3d654aefc01b`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:19,230::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:19,233::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:19,237::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:19,239::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:19,241::task::1185::TaskManager.Task::(prepare) Task=`24a2f822-d893-48b7-9bf6-3d654aefc01b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:19,243::task::595::TaskManager.Task::(_updateState) Task=`24a2f822-d893-48b7-9bf6-3d654aefc01b`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:19,245::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:19,248::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:19,252::task::990::TaskManager.Task::(_decref) Task=`24a2f822-d893-48b7-9bf6-3d654aefc01b`::ref 0 aborting False
>MainThread::DEBUG::2014-04-11 22:02:19,685::vdsm::55::vds::(sigtermHandler) Received signal 15
>MainThread::INFO::2014-04-11 22:02:19,806::vmChannels::183::vds::(stop) VM channels listener was stopped.
>MainThread::INFO::2014-04-11 22:02:19,808::momIF::84::MOM::(stop) Shutting down MOM
>MainThread::DEBUG::2014-04-11 22:02:19,810::task::595::TaskManager.Task::(_updateState) Task=`9b63cabe-d01c-48bb-9ebf-4e724b3add44`::moving from state init -> state preparing
>MainThread::INFO::2014-04-11 22:02:19,810::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None)
>Thread-11::DEBUG::2014-04-11 22:02:19,812::storageServer::698::ConnectionMonitor::(_monitorConnections) Monitoring stopped
>Thread-25::ERROR::2014-04-11 22:02:19,833::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-25::DEBUG::2014-04-11 22:02:19,836::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain b7663d70-e658-41fa-b9f0-8da83c9eddce changed its status to Invalid
>Thread-1014::DEBUG::2014-04-11 22:02:19,846::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-1014::DEBUG::2014-04-11 22:02:19,851::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-1014::DEBUG::2014-04-11 22:02:19,856::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_upgradePoolDomain`
>Thread-1014::DEBUG::2014-04-11 22:02:19,861::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>VM Channels Listener::INFO::2014-04-11 22:02:19,899::vmChannels::178::vds::(run) VM channels listener thread has ended.
>Thread-155::DEBUG::2014-04-11 22:02:20,268::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:20,273::task::595::TaskManager.Task::(_updateState) Task=`25cada56-2c55-4c65-b7d8-ab8c102cfe13`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:20,276::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:20,279::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:20,285::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:20,287::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:20,289::task::1185::TaskManager.Task::(prepare) Task=`25cada56-2c55-4c65-b7d8-ab8c102cfe13`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:20,291::task::595::TaskManager.Task::(_updateState) Task=`25cada56-2c55-4c65-b7d8-ab8c102cfe13`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:20,295::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:20,301::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:20,304::task::990::TaskManager.Task::(_decref) Task=`25cada56-2c55-4c65-b7d8-ab8c102cfe13`::ref 0 aborting False
>MainThread::DEBUG::2014-04-11 22:02:20,687::vdsm::55::vds::(sigtermHandler) Received signal 15
>MainThread::DEBUG::2014-04-11 22:02:20,687::clientIF::207::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently
>Thread-155::DEBUG::2014-04-11 22:02:21,322::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:21,325::task::595::TaskManager.Task::(_updateState) Task=`d59aa97a-fd80-4165-b867-5ef2cb5cdc44`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:21,328::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:21,332::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:21,335::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:21,336::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:21,338::task::1185::TaskManager.Task::(prepare) Task=`d59aa97a-fd80-4165-b867-5ef2cb5cdc44`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:21,339::task::595::TaskManager.Task::(_updateState) Task=`d59aa97a-fd80-4165-b867-5ef2cb5cdc44`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:21,344::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:21,346::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:21,348::task::990::TaskManager.Task::(_decref) Task=`d59aa97a-fd80-4165-b867-5ef2cb5cdc44`::ref 0 aborting False
>MainThread::DEBUG::2014-04-11 22:02:21,688::vdsm::55::vds::(sigtermHandler) Received signal 15
>MainThread::DEBUG::2014-04-11 22:02:21,688::clientIF::207::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently
>Thread-155::DEBUG::2014-04-11 22:02:21,915::task::595::TaskManager.Task::(_updateState) Task=`569c8eb1-816a-48c0-8cd7-ab185821b22f`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:21,919::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-155::INFO::2014-04-11 22:02:21,925::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '40.7', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000302115', 'lastCheck': '9.5', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '110.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.1', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '110.3', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.9', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000279867', 'lastCheck': '8.1', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:02:21,927::task::1185::TaskManager.Task::(prepare) Task=`569c8eb1-816a-48c0-8cd7-ab185821b22f`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '40.7', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000302115', 'lastCheck': '9.5', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '110.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.1', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '110.3', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.9', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000279867', 'lastCheck': '8.1', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:02:21,929::task::595::TaskManager.Task::(_updateState) Task=`569c8eb1-816a-48c0-8cd7-ab185821b22f`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:21,933::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:21,935::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:21,937::task::990::TaskManager.Task::(_decref) Task=`569c8eb1-816a-48c0-8cd7-ab185821b22f`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:22,364::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:22,367::task::595::TaskManager.Task::(_updateState) Task=`7299981d-759d-4e2b-980c-5e4d519f0a3e`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:22,369::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:22,373::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:22,378::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:22,380::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:22,381::task::1185::TaskManager.Task::(prepare) Task=`7299981d-759d-4e2b-980c-5e4d519f0a3e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:22,383::task::595::TaskManager.Task::(_updateState) Task=`7299981d-759d-4e2b-980c-5e4d519f0a3e`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:22,390::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:22,393::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:22,395::task::990::TaskManager.Task::(_decref) Task=`7299981d-759d-4e2b-980c-5e4d519f0a3e`::ref 0 aborting False
>MainThread::DEBUG::2014-04-11 22:02:22,700::vdsm::55::vds::(sigtermHandler) Received signal 15
>MainThread::DEBUG::2014-04-11 22:02:22,703::clientIF::207::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently
>Thread-22::DEBUG::2014-04-11 22:02:22,738::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:02:22,775::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000207691 s, 1.8 MB/s\n'; <rc> = 0
>MainThread::INFO::2014-04-11 22:02:23,068::domainMonitor::125::Storage.DomainMonitor::(close) Stopping domain monitors
>MainThread::INFO::2014-04-11 22:02:23,068::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-155::DEBUG::2014-04-11 22:02:23,407::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:23,408::task::595::TaskManager.Task::(_updateState) Task=`bdd239f1-2226-4b26-a150-333d6e9e367b`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:23,408::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:23,409::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:23,409::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:23,409::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:23,410::task::1185::TaskManager.Task::(prepare) Task=`bdd239f1-2226-4b26-a150-333d6e9e367b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:23,410::task::595::TaskManager.Task::(_updateState) Task=`bdd239f1-2226-4b26-a150-333d6e9e367b`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:23,410::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:23,410::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:23,411::task::990::TaskManager.Task::(_decref) Task=`bdd239f1-2226-4b26-a150-333d6e9e367b`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:02:23,861::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:02:23,878::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00024157 s, 3.1 MB/s\n'; <rc> = 0
>Thread-155::DEBUG::2014-04-11 22:02:24,416::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:24,417::task::595::TaskManager.Task::(_updateState) Task=`cbf02b2e-9801-44a9-a70d-c3730f8c5524`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:24,418::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:24,418::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:24,418::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:24,419::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:24,419::task::1185::TaskManager.Task::(prepare) Task=`cbf02b2e-9801-44a9-a70d-c3730f8c5524`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:24,420::task::595::TaskManager.Task::(_updateState) Task=`cbf02b2e-9801-44a9-a70d-c3730f8c5524`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:24,420::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:24,420::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:24,421::task::990::TaskManager.Task::(_decref) Task=`cbf02b2e-9801-44a9-a70d-c3730f8c5524`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:25,426::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:25,428::task::595::TaskManager.Task::(_updateState) Task=`2b5d84d9-ca57-4fe0-adef-957b67d0e76b`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:25,428::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:25,429::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:25,429::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:25,430::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:25,430::task::1185::TaskManager.Task::(prepare) Task=`2b5d84d9-ca57-4fe0-adef-957b67d0e76b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:25,431::task::595::TaskManager.Task::(_updateState) Task=`2b5d84d9-ca57-4fe0-adef-957b67d0e76b`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:25,431::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:25,431::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:25,432::task::990::TaskManager.Task::(_decref) Task=`2b5d84d9-ca57-4fe0-adef-957b67d0e76b`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:26,436::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:26,437::task::595::TaskManager.Task::(_updateState) Task=`03bf9766-f4c0-4eea-84b8-b8154ae294fb`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:26,437::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:26,438::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:26,438::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:26,438::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:26,439::task::1185::TaskManager.Task::(prepare) Task=`03bf9766-f4c0-4eea-84b8-b8154ae294fb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:26,439::task::595::TaskManager.Task::(_updateState) Task=`03bf9766-f4c0-4eea-84b8-b8154ae294fb`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:26,440::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:26,440::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:26,440::task::990::TaskManager.Task::(_decref) Task=`03bf9766-f4c0-4eea-84b8-b8154ae294fb`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:27,447::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:27,448::task::595::TaskManager.Task::(_updateState) Task=`5e466eb1-1d12-421d-bc56-30e286d026ce`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:27,448::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:27,449::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:27,449::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:27,449::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:27,450::task::1185::TaskManager.Task::(prepare) Task=`5e466eb1-1d12-421d-bc56-30e286d026ce`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:27,450::task::595::TaskManager.Task::(_updateState) Task=`5e466eb1-1d12-421d-bc56-30e286d026ce`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:27,450::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:27,451::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:27,452::task::990::TaskManager.Task::(_decref) Task=`5e466eb1-1d12-421d-bc56-30e286d026ce`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:28,456::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:28,457::task::595::TaskManager.Task::(_updateState) Task=`552ee523-3f64-402c-bd96-651e7f1c67be`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:28,458::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:28,458::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:28,459::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:28,459::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:28,460::task::1185::TaskManager.Task::(prepare) Task=`552ee523-3f64-402c-bd96-651e7f1c67be`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:28,460::task::595::TaskManager.Task::(_updateState) Task=`552ee523-3f64-402c-bd96-651e7f1c67be`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:28,461::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:28,461::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:28,461::task::990::TaskManager.Task::(_decref) Task=`552ee523-3f64-402c-bd96-651e7f1c67be`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:29,467::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:29,468::task::595::TaskManager.Task::(_updateState) Task=`cda0bb55-7abd-4454-8788-99fdfd784b02`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:29,468::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:29,469::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:29,469::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:29,469::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:29,471::task::1185::TaskManager.Task::(prepare) Task=`cda0bb55-7abd-4454-8788-99fdfd784b02`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:29,472::task::595::TaskManager.Task::(_updateState) Task=`cda0bb55-7abd-4454-8788-99fdfd784b02`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:29,472::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:29,472::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:29,473::task::990::TaskManager.Task::(_decref) Task=`cda0bb55-7abd-4454-8788-99fdfd784b02`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:30,480::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:30,482::task::595::TaskManager.Task::(_updateState) Task=`443f4517-e836-485e-94d3-e52c8c9573a5`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:30,482::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:30,482::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:30,483::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:30,483::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:30,484::task::1185::TaskManager.Task::(prepare) Task=`443f4517-e836-485e-94d3-e52c8c9573a5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:30,485::task::595::TaskManager.Task::(_updateState) Task=`443f4517-e836-485e-94d3-e52c8c9573a5`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:30,485::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:30,486::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:30,486::task::990::TaskManager.Task::(_decref) Task=`443f4517-e836-485e-94d3-e52c8c9573a5`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:31,492::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:31,493::task::595::TaskManager.Task::(_updateState) Task=`2c363f81-8294-4e9a-ac7b-a7cf65a58bd8`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:31,493::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:31,493::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:31,494::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:31,494::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:31,495::task::1185::TaskManager.Task::(prepare) Task=`2c363f81-8294-4e9a-ac7b-a7cf65a58bd8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:31,496::task::595::TaskManager.Task::(_updateState) Task=`2c363f81-8294-4e9a-ac7b-a7cf65a58bd8`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:31,497::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:31,497::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:31,498::task::990::TaskManager.Task::(_decref) Task=`2c363f81-8294-4e9a-ac7b-a7cf65a58bd8`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:32,503::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:32,505::task::595::TaskManager.Task::(_updateState) Task=`4775c86d-484e-4e81-9ee9-b2c69b3c7b5e`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:32,506::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:32,506::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:32,507::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:32,507::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:32,507::task::1185::TaskManager.Task::(prepare) Task=`4775c86d-484e-4e81-9ee9-b2c69b3c7b5e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:32,508::task::595::TaskManager.Task::(_updateState) Task=`4775c86d-484e-4e81-9ee9-b2c69b3c7b5e`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:32,508::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:32,509::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:32,509::task::990::TaskManager.Task::(_decref) Task=`4775c86d-484e-4e81-9ee9-b2c69b3c7b5e`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:02:32,832::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:02:32,847::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000268373 s, 1.4 MB/s\n'; <rc> = 0
>Thread-155::DEBUG::2014-04-11 22:02:33,517::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:33,520::task::595::TaskManager.Task::(_updateState) Task=`ec0f6158-d086-4957-affe-1552e341c7b6`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:33,521::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:33,522::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:33,522::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:33,522::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:33,526::task::1185::TaskManager.Task::(prepare) Task=`ec0f6158-d086-4957-affe-1552e341c7b6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:33,526::task::595::TaskManager.Task::(_updateState) Task=`ec0f6158-d086-4957-affe-1552e341c7b6`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:33,526::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:33,527::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:33,527::task::990::TaskManager.Task::(_decref) Task=`ec0f6158-d086-4957-affe-1552e341c7b6`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:02:33,937::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:02:33,951::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000257954 s, 2.9 MB/s\n'; <rc> = 0
>Thread-155::DEBUG::2014-04-11 22:02:34,536::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:34,537::task::595::TaskManager.Task::(_updateState) Task=`fd0d552f-a7ff-451d-926a-56fc9347a480`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:34,540::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:34,541::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:34,541::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:34,542::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:34,544::task::1185::TaskManager.Task::(prepare) Task=`fd0d552f-a7ff-451d-926a-56fc9347a480`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:34,544::task::595::TaskManager.Task::(_updateState) Task=`fd0d552f-a7ff-451d-926a-56fc9347a480`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:34,545::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:34,545::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:34,545::task::990::TaskManager.Task::(_decref) Task=`fd0d552f-a7ff-451d-926a-56fc9347a480`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:35,556::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:35,560::task::595::TaskManager.Task::(_updateState) Task=`03fe49be-bc7c-410d-9651-d0ccba62c999`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:35,563::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:35,565::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:35,567::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:35,569::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:35,571::task::1185::TaskManager.Task::(prepare) Task=`03fe49be-bc7c-410d-9651-d0ccba62c999`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:35,572::task::595::TaskManager.Task::(_updateState) Task=`03fe49be-bc7c-410d-9651-d0ccba62c999`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:35,574::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:35,575::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:35,578::task::990::TaskManager.Task::(_decref) Task=`03fe49be-bc7c-410d-9651-d0ccba62c999`::ref 0 aborting False
>Thread-155::DEBUG::2014-04-11 22:02:36,600::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc]
>Thread-155::DEBUG::2014-04-11 22:02:36,615::task::595::TaskManager.Task::(_updateState) Task=`7959ef3d-5634-447f-8b77-06045d526aea`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:36,624::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None)
>Thread-155::DEBUG::2014-04-11 22:02:36,633::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0
>Thread-155::DEBUG::2014-04-11 22:02:36,642::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}
>Thread-155::INFO::2014-04-11 22:02:36,652::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:36,661::task::1185::TaskManager.Task::(prepare) Task=`7959ef3d-5634-447f-8b77-06045d526aea`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}}
>Thread-155::DEBUG::2014-04-11 22:02:36,670::task::595::TaskManager.Task::(_updateState) Task=`7959ef3d-5634-447f-8b77-06045d526aea`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:36,680::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:36,687::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:36,696::task::990::TaskManager.Task::(_decref) Task=`7959ef3d-5634-447f-8b77-06045d526aea`::ref 0 aborting False
>9f4a86db-7348-40cf-aa72-ebd91cba08c0::ERROR::2014-04-11 22:02:37,158::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
>    self.checkBackupDomain(__securityOverride=True)
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 1410, in checkBackupDomain
>    if dom.isBackup():
>  File "/usr/share/vdsm/storage/sd.py", line 788, in isBackup
>    return self.getMetaParam(DMDK_CLASS) == BACKUP_DOMAIN
>  File "/usr/share/vdsm/storage/sd.py", line 722, in getMetaParam
>    return self._metadata[key]
>  File "/usr/share/vdsm/storage/persistentDict.py", line 89, in __getitem__
>    return dec(self._dict[key])
>  File "/usr/share/vdsm/storage/persistentDict.py", line 199, in __getitem__
>    with self._accessWrapper():
>  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
>    return self.gen.next()
>  File "/usr/share/vdsm/storage/persistentDict.py", line 152, in _accessWrapper
>    self.refresh()
>  File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
>    lines = self._metaRW.readlines()
>  File "/usr/share/vdsm/storage/fileSD.py", line 108, in readlines
>    return stripNewLines(self._oop.directReadLines(self._metafile))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-155::DEBUG::2014-04-11 22:02:37,396::task::595::TaskManager.Task::(_updateState) Task=`b4384bc6-414f-4e31-8119-6c98b49b3ec2`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:37,413::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-155::INFO::2014-04-11 22:02:37,456::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '56.2', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000268373', 'lastCheck': '4.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '125.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '17.6', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '125.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.4', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000257954', 'lastCheck': '3.5', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:02:37,469::task::1185::TaskManager.Task::(prepare) Task=`b4384bc6-414f-4e31-8119-6c98b49b3ec2`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '56.2', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000268373', 'lastCheck': '4.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '125.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '17.6', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '125.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.4', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000257954', 'lastCheck': '3.5', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:02:37,483::task::595::TaskManager.Task::(_updateState) Task=`b4384bc6-414f-4e31-8119-6c98b49b3ec2`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:37,496::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:37,509::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:37,525::task::990::TaskManager.Task::(_decref) Task=`b4384bc6-414f-4e31-8119-6c98b49b3ec2`::ref 0 aborting False
>Thread-994::WARNING::2014-04-11 22:02:37,974::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-994::DEBUG::2014-04-11 22:02:38,001::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-996::WARNING::2014-04-11 22:02:37,999::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-995::WARNING::2014-04-11 22:02:38,005::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-996::DEBUG::2014-04-11 22:02:38,019::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-995::DEBUG::2014-04-11 22:02:38,025::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-31::ERROR::2014-04-11 22:02:38,036::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-31::ERROR::2014-04-11 22:02:38,045::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-1001::WARNING::2014-04-11 22:02:38,319::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-1001::DEBUG::2014-04-11 22:02:38,326::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-1003::WARNING::2014-04-11 22:02:38,342::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-1002::WARNING::2014-04-11 22:02:38,343::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-1003::DEBUG::2014-04-11 22:02:38,346::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-1002::DEBUG::2014-04-11 22:02:38,347::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-38::ERROR::2014-04-11 22:02:38,349::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-38::ERROR::2014-04-11 22:02:38,349::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-22::DEBUG::2014-04-11 22:02:42,879::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:02:42,891::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000293462 s, 1.3 MB/s\n'; <rc> = 0
>Thread-44::DEBUG::2014-04-11 22:02:43,989::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:02:43,999::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000218867 s, 3.4 MB/s\n'; <rc> = 0
>Thread-31::DEBUG::2014-04-11 22:02:48,056::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-31::DEBUG::2014-04-11 22:02:48,056::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-31::DEBUG::2014-04-11 22:02:48,057::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-31::DEBUG::2014-04-11 22:02:48,057::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-31::DEBUG::2014-04-11 22:02:48,058::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-31::DEBUG::2014-04-11 22:02:48,059::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-31::DEBUG::2014-04-11 22:02:48,113::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-31::DEBUG::2014-04-11 22:02:48,114::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-31::DEBUG::2014-04-11 22:02:48,255::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-31::DEBUG::2014-04-11 22:02:48,256::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-31::DEBUG::2014-04-11 22:02:48,257::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-31::DEBUG::2014-04-11 22:02:48,257::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-31::DEBUG::2014-04-11 22:02:48,258::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-31::DEBUG::2014-04-11 22:02:48,258::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-31::DEBUG::2014-04-11 22:02:48,259::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-31::DEBUG::2014-04-11 22:02:48,259::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-31::ERROR::2014-04-11 22:02:48,260::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-31::ERROR::2014-04-11 22:02:48,260::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-31::DEBUG::2014-04-11 22:02:48,261::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-31::DEBUG::2014-04-11 22:02:48,264::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-38::ERROR::2014-04-11 22:02:48,350::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-38::ERROR::2014-04-11 22:02:48,351::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-38::DEBUG::2014-04-11 22:02:48,353::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-38::DEBUG::2014-04-11 22:02:48,355::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-31::DEBUG::2014-04-11 22:02:48,618::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-31::WARNING::2014-04-11 22:02:48,623::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-38::DEBUG::2014-04-11 22:02:48,700::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-38::WARNING::2014-04-11 22:02:48,704::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-38::DEBUG::2014-04-11 22:02:48,705::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::ERROR::2014-04-11 22:02:51,282::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-20::DEBUG::2014-04-11 22:02:51,282::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-20::INFO::2014-04-11 22:02:51,283::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-20::DEBUG::2014-04-11 22:02:51,284::domainMonitor::184::Storage.DomainMonitorThread::(_monitorLoop) Unable to release the host id 1 for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 181, in _monitorLoop
>    self.domain.releaseHostId(self.hostId, unused=True)
>  File "/usr/share/vdsm/storage/sd.py", line 461, in releaseHostId
>    self._clusterLock.releaseHostId(hostId, async, unused)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 214, in releaseHostId
>    raise se.ReleaseHostIdFailure(self._sdUUID, e)
>ReleaseHostIdFailure: Cannot release host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(115, 'Sanlock lockspace remove failure', 'Operation now in progress'))
>MainThread::INFO::2014-04-11 22:02:51,285::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-22::DEBUG::2014-04-11 22:02:51,285::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>MainThread::INFO::2014-04-11 22:02:51,286::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-155::DEBUG::2014-04-11 22:02:53,262::task::595::TaskManager.Task::(_updateState) Task=`da557d9e-4d12-427a-b1f6-814981209639`::moving from state init -> state preparing
>Thread-155::INFO::2014-04-11 22:02:53,262::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-155::INFO::2014-04-11 22:02:53,263::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '15.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.4', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '50.3', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000218867', 'lastCheck': '9.3', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:02:53,263::task::1185::TaskManager.Task::(prepare) Task=`da557d9e-4d12-427a-b1f6-814981209639`::finished: {'3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '15.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.4', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '50.3', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000218867', 'lastCheck': '9.3', 'valid': True}}
>Thread-155::DEBUG::2014-04-11 22:02:53,264::task::595::TaskManager.Task::(_updateState) Task=`da557d9e-4d12-427a-b1f6-814981209639`::moving from state preparing -> state finished
>Thread-155::DEBUG::2014-04-11 22:02:53,264::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-155::DEBUG::2014-04-11 22:02:53,265::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-155::DEBUG::2014-04-11 22:02:53,265::task::990::TaskManager.Task::(_decref) Task=`da557d9e-4d12-427a-b1f6-814981209639`::ref 0 aborting False
>Thread-44::DEBUG::2014-04-11 22:02:54,005::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 21619c8e-99ea-4813-be02-d708971e5393
>Thread-44::DEBUG::2014-04-11 22:02:54,082::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393
>Thread-44::DEBUG::2014-04-11 22:02:54,085::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-44::DEBUG::2014-04-11 22:02:54,101::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2']
>Thread-44::DEBUG::2014-04-11 22:02:54,104::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-44::INFO::2014-04-11 22:02:54,105::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 21619c8e-99ea-4813-be02-d708971e5393_imageNS already registered
>Thread-44::INFO::2014-04-11 22:02:54,105::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 21619c8e-99ea-4813-be02-d708971e5393_volumeNS already registered
>Thread-44::DEBUG::2014-04-11 22:02:54,138::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-44::DEBUG::2014-04-11 22:02:54,151::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000261909 s, 2.9 MB/s\n'; <rc> = 0
>MainThread::INFO::2014-04-11 22:03:00,061::vdsm::119::vds::(run) (PID: 33265) I am the actual vdsm 4.14.6-12.git49cacf0.el6 ashtivh01.suprtekstic.com (2.6.32-431.11.2.el6.x86_64)
>MainThread::DEBUG::2014-04-11 22:03:00,062::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage'
>MainThread::DEBUG::2014-04-11 22:03:00,063::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
>MainThread::WARNING::2014-04-11 22:03:00,070::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists
>MainThread::DEBUG::2014-04-11 22:03:01,623::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
>MainThread::DEBUG::2014-04-11 22:03:01,650::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving
>MainThread::DEBUG::2014-04-11 22:03:01,650::hsm::424::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
>MainThread::DEBUG::2014-04-11 22:03:01,931::hsm::424::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
>MainThread::DEBUG::2014-04-11 22:03:01,931::hsm::448::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
>MainThread::DEBUG::2014-04-11 22:03:01,941::hsm::480::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
>MainThread::DEBUG::2014-04-11 22:03:01,941::hsm::481::Storage.HSM::(__cleanStorageRepository) Mount list: ['/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage', '/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage', '/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage', '/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso', '/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage']
>MainThread::DEBUG::2014-04-11 22:03:01,941::hsm::483::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
>MainThread::DEBUG::2014-04-11 22:03:01,946::hsm::526::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
>storageRefresh::DEBUG::2014-04-11 22:03:01,947::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>MainThread::INFO::2014-04-11 22:03:01,949::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
>storageRefresh::DEBUG::2014-04-11 22:03:01,950::misc::764::SamplingMethod::(__call__) Got in to sampling method
>storageRefresh::DEBUG::2014-04-11 22:03:02,005::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>storageRefresh::DEBUG::2014-04-11 22:03:02,008::misc::764::SamplingMethod::(__call__) Got in to sampling method
>storageRefresh::DEBUG::2014-04-11 22:03:02,009::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>storageRefresh::DEBUG::2014-04-11 22:03:02,012::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:03:02,032::misc::772::SamplingMethod::(__call__) Returning last result
>storageRefresh::DEBUG::2014-04-11 22:03:02,032::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>MainThread::DEBUG::2014-04-11 22:03:02,053::task::595::TaskManager.Task::(_updateState) Task=`20ba9a8b-8c3d-4503-9f6a-c7cf0e5b4607`::moving from state init -> state preparing
>MainThread::INFO::2014-04-11 22:03:02,054::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x7f4eb80799e0>>)
>MainThread::INFO::2014-04-11 22:03:02,054::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
>MainThread::DEBUG::2014-04-11 22:03:02,055::task::1185::TaskManager.Task::(prepare) Task=`20ba9a8b-8c3d-4503-9f6a-c7cf0e5b4607`::finished: None
>MainThread::DEBUG::2014-04-11 22:03:02,055::task::595::TaskManager.Task::(_updateState) Task=`20ba9a8b-8c3d-4503-9f6a-c7cf0e5b4607`::moving from state preparing -> state finished
>MainThread::DEBUG::2014-04-11 22:03:02,055::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>MainThread::DEBUG::2014-04-11 22:03:02,056::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>MainThread::DEBUG::2014-04-11 22:03:02,056::task::990::TaskManager.Task::(_decref) Task=`20ba9a8b-8c3d-4503-9f6a-c7cf0e5b4607`::ref 0 aborting False
>MainThread::INFO::2014-04-11 22:03:02,071::momIF::47::MOM::(__init__) Starting up MOM
>MainThread::INFO::2014-04-11 22:03:02,079::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds.
>VM Channels Listener::INFO::2014-04-11 22:03:02,102::vmChannels::170::vds::(run) Starting VM channels listener thread.
>clientIFinit::DEBUG::2014-04-11 22:03:02,103::libvirtconnection::145::root::(get) trying to connect libvirt
>MainThread::WARNING::2014-04-11 22:03:02,110::clientIF::181::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
>storageRefresh::DEBUG::2014-04-11 22:03:02,162::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:03:02,163::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,174::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,174::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,175::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,175::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,175::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,176::misc::772::SamplingMethod::(__call__) Returning last result
>storageRefresh::DEBUG::2014-04-11 22:03:02,176::lvm::324::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,179::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:03:02,509::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:03:02,513::lvm::349::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,517::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,526::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:03:02,785::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:03:02,786::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:02,787::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:03:03,036::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:03:03,049::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:03:03,050::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:03:03,299::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:03:03,299::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>storageRefresh::WARNING::2014-04-11 22:03:03,300::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists
>storageRefresh::DEBUG::2014-04-11 22:03:03,300::hsm::392::Storage.HSM::(storageRefresh) HSM is ready
>Thread-13::DEBUG::2014-04-11 22:03:05,731::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getCapabilities with () {} flowID [13a209e1]
>Thread-13::DEBUG::2014-04-11 22:03:05,841::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-13::DEBUG::2014-04-11 22:03:05,857::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:03:05,905::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.6' (cwd None)
>Thread-13::DEBUG::2014-04-11 22:03:05,915::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:03:05,921::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None)
>Thread-13::DEBUG::2014-04-11 22:03:05,930::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:03:05,937::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.5' (cwd None)
>Thread-13::DEBUG::2014-04-11 22:03:05,945::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:03:05,950::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-13::DEBUG::2014-04-11 22:03:05,958::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-13::DEBUG::2014-04-11 22:03:06,001::caps::458::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-13::DEBUG::2014-04-11 22:03:06,001::caps::458::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-13::DEBUG::2014-04-11 22:03:06,003::caps::458::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-13::DEBUG::2014-04-11 22:03:06,011::caps::458::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-13::DEBUG::2014-04-11 22:03:06,012::caps::458::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-13::DEBUG::2014-04-11 22:03:06,012::caps::458::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-13::DEBUG::2014-04-11 22:03:06,022::caps::458::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-13::DEBUG::2014-04-11 22:03:06,023::caps::458::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-13::DEBUG::2014-04-11 22:03:06,023::caps::458::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-13::DEBUG::2014-04-11 22:03:06,024::caps::458::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-13::DEBUG::2014-04-11 22:03:06,024::caps::458::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-13::DEBUG::2014-04-11 22:03:06,025::caps::458::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-13::DEBUG::2014-04-11 22:03:06,050::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:93188a588127'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395791995.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '12.git49cacf0.el6', 'buildtime': 1397150575L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz', 'hooks': {'after_vm_start': {'50_promisc': {'md5': '5ea469fa06dd7cd1e52508672a7519db'}}, 'before_vm_migrate_source': {'50_sriov': {'md5': '47a59103b7c8b1439ae17dcd5ed197cd'}, '50_hostusb': {'md5': 'e50a0975acd8bafd0052b65a787c2fdd'}, '50_scratchpad': {'md5': 'be90995f959c60c5110c6a0bef67c7b8'}, '50_floppy': {'md5': 'c68fb91d44105c8b6f2d80566314027a'}}, 'after_device_destroy': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_vm_destroy': {'50_promisc': {'md5': 'dae0434ea8ca2492422a6fde5cc47a1e'}}, 'before_vm_start': {'50_sriov': {'md5': '3ebc60cd2e4eb089820102285fad7c45'}, '50_vmfex': {'md5': '9f5abb892ddb6b3daa779985d38d9f55'}, '50_qos': {'md5': 'cdf6f89d277a886691812ce1ec2fe105'}, '50_scratchpad': {'md5': '178171eb62ba95fc548e2450013f5ad5'}, '50_numa': {'md5': '5008c2826714ac5b63748780aabd2f25'}, '50_floppy': {'md5': '202fe18705a7d4c50c40c126e8f8dbe8'}, '50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}, '50_pincpu': {'md5': '0b5fb99ff0e7acb9ad534b87c02c59e3'}, '50_hugepages': {'md5': '6713d703f9d316f1664145b700cd7904'}, '50_fileinject': {'md5': '290dafd45601e010e0f496fa5b889a76'}, '50_macspoof': {'md5': '25deea559772719b90117f2e1009d823'}, '60_checkimages': {'md5': 'abd32c723a90ab2816e2163f5e02faf7'}, '10_faqemu': {'md5': 'e3cb338be6b9075a04bed0e7400ab7e2'}, '50_directlun': {'md5': 'dba5d80fdb7119ee0f4a40c26eac8448'}, '50_smbios': {'md5': '1a6bd8eddc3a86943e558bc4d3e73bd6'}, '50_vmdisk': {'md5': '67617c6e2d796388ddc6c065a97f714e'}, '50_hostusb': {'md5': '21c58568f3eae4f0d091005e0eff891a'}, '50_isolatedprivatevlan': {'md5': '0a25d7cd0b807ea5e0dfc3318ee98c91'}}, 'after_device_migrate_destination': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'after_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_vm_migrate_destination': {'50_vmfex': {'md5': 'eac2633f47fb309b4a5b7fe76531baec'}, '50_hugepages': {'md5': '6ddc484746297308b9f9acc5fcabaf10'}, '50_directlun': {'md5': '8a29ce41de079630cb58d48e280d55c2'}}, 'after_nic_hotunplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_vm_destroy': {'50_sriov': {'md5': '5890b2899d450495a62f3611fad3c526'}, '50_directlun': {'md5': '6d05c19e68a141eeb7ee5981d68c0cfa'}, '50_hostusb': {'md5': '0a37bf0a26546e3ab5e424b13653ce2b'}, '50_hugepages': {'md5': '4a980b78f8ee908bf2c61ea9bf156f4a'}, '50_scratchpad': {'md5': '316d21a669f97e83b485b45ab0b23eee'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0', 'addr': '172.16.10.1', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'bridged': False, 'interface': u'bond0', 'ipv6gateway': '::', 'gateway': '172.16.0.1'}, 'Blue': {'iface': 'Blue', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.6']}, 'Infrastructure': {'iface': 'Infrastructure', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.4']}, 'migration': {'iface': u'bond0.5', 'addr': '', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'bridged': False, 'interface': u'bond0.5', 'ipv6gateway': '::', 'gateway': ''}, 'Red': {'iface': 'Red', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.3']}}, 'bridges': {'Blue': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.6']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'Infrastructure': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.4']}, 'Red': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.3']}}, 'uuid': '4C4C4544-0059-5310-8031-B9C04F445A31', 'lastClientIface': 'bond0', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:5F', 'MASTER': 'bond0', 'DEVICE': 'em1', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:5F', 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:60', 'MASTER': 'bond0', 'DEVICE': 'em2', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:60', 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '12', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,lahf_lm,ida,arat,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:93188a588127', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '255.255.0.0', 'addr': '172.16.10.1', 'slaves': ['em1', 'em2'], 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.10.1', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond0', 'MTU': '1500', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32057', 'cpuSpeed': '1900.023', 'version_name': 'Snow Man', 'vlans': {'bond0.3': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Red', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 3, 'mtu': '1500'}, 'bond0.4': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Infrastructure', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 4, 'mtu': '1500'}, 'bond0.5': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'DEVICE': 'bond0.5', 'NM_CONTROLLED': 'no', 'VLAN': 'yes', 'DEFROUTE': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 5, 'mtu': '1500'}, 'bond0.6': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Blue', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.6', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 6, 'mtu': '1500'}}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '24', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.10.1'}}
>Thread-13::DEBUG::2014-04-11 22:03:06,090::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getHardwareInfo with () {} flowID [13a209e1]
>Thread-13::DEBUG::2014-04-11 22:03:06,097::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R420', 'systemSerialNumber': '9YS1DZ1', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0059-5310-8031-B9C04F445A31', 'systemManufacturer': 'Dell Inc.'}}
>Thread-13::DEBUG::2014-04-11 22:03:06,278::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [5757a999]
>Thread-13::DEBUG::2014-04-11 22:03:06,279::task::595::TaskManager.Task::(_updateState) Task=`b967b538-bfc8-4542-b173-e0038dad875f`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:03:06,280::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', conList=[{'connection': 'ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c', 'port': ''}, {'connection': 'ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/exports', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54', 'port': ''}, {'connection': 'ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'deb06d0b-14c6-490e-8fbb-552861511109', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/infrastructure', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac', 'port': ''}], options=None)
>Thread-13::DEBUG::2014-04-11 22:03:06,288::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage
>Thread-14::DEBUG::2014-04-11 22:03:09,742::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [61b945bb]
>Thread-14::DEBUG::2014-04-11 22:03:09,745::task::595::TaskManager.Task::(_updateState) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:03:09,745::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:03:09,745::task::866::TaskManager.Task::(_setError) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:03:09,749::task::885::TaskManager.Task::(_run) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::Task._run: 483c539f-ff25-4895-ad9c-b51fdd90eb02 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:03:09,750::task::1211::TaskManager.Task::(stop) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:03:09,750::task::990::TaskManager.Task::(_decref) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:03:09,750::task::1168::TaskManager.Task::(prepare) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:03:09,751::task::1173::TaskManager.Task::(prepare) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:03:09,751::task::990::TaskManager.Task::(_decref) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:03:09,751::task::925::TaskManager.Task::(_doAbort) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:03:09,751::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:03:09,752::task::595::TaskManager.Task::(_updateState) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:03:09,752::task::550::TaskManager.Task::(__state_aborting) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:03:09,752::task::595::TaskManager.Task::(_updateState) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:03:09,753::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:03:09,753::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:03:09,753::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:03:09,798::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [61b945bb]
>Thread-14::DEBUG::2014-04-11 22:03:09,798::task::595::TaskManager.Task::(_updateState) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:03:09,799::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-14::ERROR::2014-04-11 22:03:09,799::task::866::TaskManager.Task::(_setError) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
>    raise se.SpmStatusError()
>SpmStatusError: Not SPM: ()
>Thread-14::DEBUG::2014-04-11 22:03:09,799::task::885::TaskManager.Task::(_run) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::Task._run: 6ea71689-0a3f-49da-8d73-f0a04c2a9144 () {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:03:09,800::task::1211::TaskManager.Task::(stop) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:03:09,800::task::990::TaskManager.Task::(_decref) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:03:09,800::task::1168::TaskManager.Task::(prepare) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::aborting: Task is aborted: 'Not SPM' - code 654
>Thread-14::DEBUG::2014-04-11 22:03:09,801::task::1173::TaskManager.Task::(prepare) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::Prepare: aborted: Not SPM
>Thread-14::DEBUG::2014-04-11 22:03:09,801::task::990::TaskManager.Task::(_decref) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:03:09,801::task::925::TaskManager.Task::(_doAbort) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:03:09,801::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:03:09,802::task::595::TaskManager.Task::(_updateState) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:03:09,802::task::550::TaskManager.Task::(__state_aborting) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:03:09,802::task::595::TaskManager.Task::(_updateState) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:03:09,802::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:03:09,803::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:03:09,803::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
>Thread-13::DEBUG::2014-04-11 22:04:06,298::hsm::2387::Storage.HSM::(connectStorageServer) prefetch failed: {}
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2384, in connectStorageServer
>    doms = self.__prefetchDomains(domType, conObj)
>  File "/usr/share/vdsm/storage/hsm.py", line 2321, in __prefetchDomains
>    goop.glob.glob(os.path.join(lPath, uuidPatern)))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-13::DEBUG::2014-04-11 22:04:06,299::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {}
>Thread-13::DEBUG::2014-04-11 22:04:06,305::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage
>Thread-13::DEBUG::2014-04-11 22:05:06,315::hsm::2387::Storage.HSM::(connectStorageServer) prefetch failed: {}
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2384, in connectStorageServer
>    doms = self.__prefetchDomains(domType, conObj)
>  File "/usr/share/vdsm/storage/hsm.py", line 2321, in __prefetchDomains
>    goop.glob.glob(os.path.join(lPath, uuidPatern)))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-13::DEBUG::2014-04-11 22:05:06,316::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {}
>Thread-13::DEBUG::2014-04-11 22:05:06,320::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/exports /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_exports' (cwd None)
>Thread-13::ERROR::2014-04-11 22:05:58,706::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-13::ERROR::2014-04-11 22:05:58,720::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-13::DEBUG::2014-04-11 22:05:58,720::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {}
>Thread-13::DEBUG::2014-04-11 22:05:58,728::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage
>Thread-14::DEBUG::2014-04-11 22:06:06,349::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:06:06,350::task::595::TaskManager.Task::(_updateState) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:06:06,350::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:06:06,351::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`339a9a3f-9611-4ba8-be51-36469ce057f9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:06:06,351::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:06:06,352::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:06:06,352::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`339a9a3f-9611-4ba8-be51-36469ce057f9`::Granted request
>Thread-14::DEBUG::2014-04-11 22:06:06,353::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`409e9708-1f83-4ae3-a368-59235998ce7f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:06:06,354::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:06:06,354::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:06:06,354::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`409e9708-1f83-4ae3-a368-59235998ce7f`::Granted request
>Thread-14::INFO::2014-04-11 22:06:06,355::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c0b7950
>Thread-14::INFO::2014-04-11 22:06:06,355::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36)
>Thread-14::DEBUG::2014-04-11 22:06:06,356::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,356::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,357::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,357::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,357::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,358::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,358::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:06:06,358::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:06:06,358::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 22:06:06,359::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:06:06,359::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 22:06:06,360::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:06:06,392::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:06:06,393::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:06:06,549::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:06:06,566::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,575::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,575::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,576::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,576::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,577::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,577::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::ERROR::2014-04-11 22:06:06,578::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::ERROR::2014-04-11 22:06:06,578::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:06:06,578::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:06:06,581::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:06:06,928::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:06:06,930::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-14::DEBUG::2014-04-11 22:06:06,930::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-13::DEBUG::2014-04-11 22:06:58,738::hsm::2387::Storage.HSM::(connectStorageServer) prefetch failed: {}
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2384, in connectStorageServer
>    doms = self.__prefetchDomains(domType, conObj)
>  File "/usr/share/vdsm/storage/hsm.py", line 2321, in __prefetchDomains
>    goop.glob.glob(os.path.join(lPath, uuidPatern)))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-13::DEBUG::2014-04-11 22:06:58,739::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {}
>Thread-13::DEBUG::2014-04-11 22:06:58,755::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/infrastructure /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure' (cwd None)
>Thread-15::WARNING::2014-04-11 22:07:06,985::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-15::DEBUG::2014-04-11 22:07:07,004::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-16::WARNING::2014-04-11 22:07:07,002::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-16::DEBUG::2014-04-11 22:07:07,011::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-17::WARNING::2014-04-11 22:07:07,016::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-14::ERROR::2014-04-11 22:07:07,018::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',)
>Thread-14::DEBUG::2014-04-11 22:07:07,019::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:07:07,020::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:07:07,020::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:07:07,021::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 22:07:07,021::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 22:07:07,021::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:07:07,022::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:07:07,022::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-14::ERROR::2014-04-11 22:07:07,022::task::866::TaskManager.Task::(_setError) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool
>    spUUID, hostID, msdUUID, masterVersion, domainsMap)
>  File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool
>    res = pool.connect(hostID, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 625, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild
>    self.setMasterDomain(msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain
>    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
>StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'
>Thread-14::DEBUG::2014-04-11 22:07:07,024::task::885::TaskManager.Task::(_run) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::Task._run: 2d141f7a-5323-436b-a21a-7bf94b55ce7d ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:07:07,024::task::1211::TaskManager.Task::(stop) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:07:07,025::task::990::TaskManager.Task::(_decref) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:07:07,025::task::1168::TaskManager.Task::(prepare) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::aborting: Task is aborted: 'Cannot find master domain' - code 304
>Thread-14::DEBUG::2014-04-11 22:07:07,025::task::1173::TaskManager.Task::(prepare) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::Prepare: aborted: Cannot find master domain
>Thread-14::DEBUG::2014-04-11 22:07:07,026::task::990::TaskManager.Task::(_decref) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:07:07,026::task::925::TaskManager.Task::(_doAbort) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:07:07,026::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:07:07,027::task::595::TaskManager.Task::(_updateState) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:07:07,027::task::550::TaskManager.Task::(__state_aborting) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:07:07,027::task::595::TaskManager.Task::(_updateState) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:07:07,027::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:07:07,028::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:07:07,028::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}}
>Thread-14::DEBUG::2014-04-11 22:07:07,054::task::595::TaskManager.Task::(_updateState) Task=`830a8c6f-5f48-44ee-9f03-0d789daef4a8`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:07:07,055::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:07:07,055::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-14::DEBUG::2014-04-11 22:07:07,055::task::1185::TaskManager.Task::(prepare) Task=`830a8c6f-5f48-44ee-9f03-0d789daef4a8`::finished: {}
>Thread-14::DEBUG::2014-04-11 22:07:07,055::task::595::TaskManager.Task::(_updateState) Task=`830a8c6f-5f48-44ee-9f03-0d789daef4a8`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:07:07,056::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:07:07,056::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:07:07,056::task::990::TaskManager.Task::(_decref) Task=`830a8c6f-5f48-44ee-9f03-0d789daef4a8`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:07:07,793::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call setMOMPolicyParameters with ({'balloonEnabled': True, 'ksmEnabled': True},) {} flowID [5757a999]
>Thread-14::DEBUG::2014-04-11 22:07:07,889::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
>Thread-14::DEBUG::2014-04-11 22:07:10,106::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4bd6819a]
>Thread-14::DEBUG::2014-04-11 22:07:10,124::task::595::TaskManager.Task::(_updateState) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:07:10,124::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:07:10,124::task::866::TaskManager.Task::(_setError) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:07:10,125::task::885::TaskManager.Task::(_run) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::Task._run: 82312493-60c4-4805-82f6-c4ddd861a537 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:07:10,125::task::1211::TaskManager.Task::(stop) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:07:10,125::task::990::TaskManager.Task::(_decref) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:07:10,126::task::1168::TaskManager.Task::(prepare) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:07:10,126::task::1173::TaskManager.Task::(prepare) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:07:10,126::task::990::TaskManager.Task::(_decref) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:07:10,127::task::925::TaskManager.Task::(_doAbort) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:07:10,127::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:07:10,127::task::595::TaskManager.Task::(_updateState) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:07:10,127::task::550::TaskManager.Task::(__state_aborting) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:07:10,128::task::595::TaskManager.Task::(_updateState) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:07:10,128::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:07:10,128::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:07:10,128::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:07:10,172::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4bd6819a]
>Thread-14::DEBUG::2014-04-11 22:07:10,173::task::595::TaskManager.Task::(_updateState) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:07:10,173::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:07:10,174::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7c1e1b75-f19a-4cae-b0ac-c75dca994ad1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:07:10,174::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:07:10,175::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:07:10,175::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7c1e1b75-f19a-4cae-b0ac-c75dca994ad1`::Granted request
>Thread-14::DEBUG::2014-04-11 22:07:10,176::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`1c59fff7-04d8-4b73-891a-7bbc2c459a94`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:07:10,176::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:07:10,176::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:07:10,177::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`1c59fff7-04d8-4b73-891a-7bbc2c459a94`::Granted request
>Thread-14::INFO::2014-04-11 22:07:10,177::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e840aec20
>Thread-14::INFO::2014-04-11 22:07:10,178::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36)
>Thread-14::DEBUG::2014-04-11 22:07:10,178::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,178::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,179::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,179::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,180::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,180::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,180::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:07:10,180::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:07:10,181::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 22:07:10,181::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:07:10,181::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 22:07:10,182::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:07:10,208::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:07:10,209::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:07:10,344::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:07:10,345::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,353::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,354::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,354::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,355::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,355::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,356::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::ERROR::2014-04-11 22:07:10,356::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::ERROR::2014-04-11 22:07:10,356::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:07:10,357::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:07:10,360::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:07:10,717::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:07:10,719::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-14::DEBUG::2014-04-11 22:07:10,720::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:07:11,049::task::595::TaskManager.Task::(_updateState) Task=`debd47f8-c976-4371-b40e-59f43a870074`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:07:11,049::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:07:11,050::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:07:11,050::task::1185::TaskManager.Task::(prepare) Task=`debd47f8-c976-4371-b40e-59f43a870074`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:07:11,050::task::595::TaskManager.Task::(_updateState) Task=`debd47f8-c976-4371-b40e-59f43a870074`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:07:11,051::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:07:11,051::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:07:11,051::task::990::TaskManager.Task::(_decref) Task=`debd47f8-c976-4371-b40e-59f43a870074`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:07:26,386::task::595::TaskManager.Task::(_updateState) Task=`2a2785ff-ef1f-4b6f-9bb4-27854056fc36`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:07:26,398::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:07:26,399::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:07:26,399::task::1185::TaskManager.Task::(prepare) Task=`2a2785ff-ef1f-4b6f-9bb4-27854056fc36`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:07:26,399::task::595::TaskManager.Task::(_updateState) Task=`2a2785ff-ef1f-4b6f-9bb4-27854056fc36`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:07:26,400::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:07:26,400::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:07:26,400::task::990::TaskManager.Task::(_decref) Task=`2a2785ff-ef1f-4b6f-9bb4-27854056fc36`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:07:41,707::task::595::TaskManager.Task::(_updateState) Task=`bc860597-8031-4590-8e9f-01687949ad49`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:07:41,707::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:07:41,708::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:07:41,708::task::1185::TaskManager.Task::(prepare) Task=`bc860597-8031-4590-8e9f-01687949ad49`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:07:41,709::task::595::TaskManager.Task::(_updateState) Task=`bc860597-8031-4590-8e9f-01687949ad49`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:07:41,709::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:07:41,709::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:07:41,710::task::990::TaskManager.Task::(_decref) Task=`bc860597-8031-4590-8e9f-01687949ad49`::ref 0 aborting False
>Thread-13::ERROR::2014-04-11 22:07:51,049::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-13::ERROR::2014-04-11 22:07:51,062::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-13::DEBUG::2014-04-11 22:07:51,062::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {}
>Thread-13::DEBUG::2014-04-11 22:07:51,074::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso
>Thread-13::DEBUG::2014-04-11 22:07:51,372::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',)
>Thread-13::DEBUG::2014-04-11 22:07:51,373::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain}
>Thread-13::DEBUG::2014-04-11 22:07:51,382::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage
>Thread-13::DEBUG::2014-04-11 22:07:51,385::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('21619c8e-99ea-4813-be02-d708971e5393',)
>Thread-13::DEBUG::2014-04-11 22:07:51,385::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain}
>Thread-13::INFO::2014-04-11 22:07:51,386::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]}
>Thread-13::DEBUG::2014-04-11 22:07:51,386::task::1185::TaskManager.Task::(prepare) Task=`b967b538-bfc8-4542-b173-e0038dad875f`::finished: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]}
>Thread-13::DEBUG::2014-04-11 22:07:51,386::task::595::TaskManager.Task::(_updateState) Task=`b967b538-bfc8-4542-b173-e0038dad875f`::moving from state preparing -> state finished
>Thread-13::DEBUG::2014-04-11 22:07:51,387::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:07:51,387::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:07:51,388::task::990::TaskManager.Task::(_decref) Task=`b967b538-bfc8-4542-b173-e0038dad875f`::ref 0 aborting False
>Thread-13::ERROR::2014-04-11 22:07:51,389::SecureXMLRPCServer::168::root::(handle_error) client ('172.16.10.1', 59605)
>Traceback (most recent call last):
>  File "/usr/lib64/python2.6/SocketServer.py", line 570, in process_request_thread
>    self.finish_request(request, client_address)
>  File "/usr/lib64/python2.6/SocketServer.py", line 332, in finish_request
>    self.RequestHandlerClass(request, client_address, self)
>  File "/usr/lib64/python2.6/SocketServer.py", line 627, in __init__
>    self.handle()
>  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 331, in handle
>    self.handle_one_request()
>  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
>    method()
>  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 219, in do_POST
>    self.send_response(200)
>  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 384, in send_response
>    self.send_header('Server', self.version_string())
>  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 390, in send_header
>    self.wfile.write("%s: %s\r\n" % (keyword, value))
>  File "/usr/lib64/python2.6/socket.py", line 324, in write
>    self.flush()
>  File "/usr/lib64/python2.6/socket.py", line 303, in flush
>    self._sock.sendall(buffer(data, write_offset, buffer_size))
>  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 222, in write
>    return self._write_bio(data)
>  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 205, in _write_bio
>    return m2.ssl_write(self.ssl, data, self._timeout)
>SSLError: (32, 'Broken pipe')
>Thread-28::DEBUG::2014-04-11 22:07:57,026::task::595::TaskManager.Task::(_updateState) Task=`4e7aedfa-64a4-48ab-8c05-e5c5ef19b2c7`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:07:57,027::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:07:57,027::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:07:57,027::task::1185::TaskManager.Task::(prepare) Task=`4e7aedfa-64a4-48ab-8c05-e5c5ef19b2c7`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:07:57,028::task::595::TaskManager.Task::(_updateState) Task=`4e7aedfa-64a4-48ab-8c05-e5c5ef19b2c7`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:07:57,028::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:07:57,029::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:07:57,029::task::990::TaskManager.Task::(_decref) Task=`4e7aedfa-64a4-48ab-8c05-e5c5ef19b2c7`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:07:59,493::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [70476e89]
>Thread-28::DEBUG::2014-04-11 22:07:59,493::task::595::TaskManager.Task::(_updateState) Task=`df78d592-6a16-4082-9277-2a886613ef82`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:07:59,494::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c', 'port': ''}], options=None)
>Thread-28::DEBUG::2014-04-11 22:07:59,501::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage
>Thread-21::WARNING::2014-04-11 22:08:10,773::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-21::DEBUG::2014-04-11 22:08:10,779::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-22::WARNING::2014-04-11 22:08:10,779::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-23::WARNING::2014-04-11 22:08:10,783::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-22::DEBUG::2014-04-11 22:08:10,787::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-23::DEBUG::2014-04-11 22:08:10,788::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-14::ERROR::2014-04-11 22:08:10,791::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',)
>Thread-14::DEBUG::2014-04-11 22:08:10,793::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:08:10,793::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:08:10,793::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:08:10,794::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 22:08:10,794::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 22:08:10,795::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:08:10,795::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:08:10,795::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-14::ERROR::2014-04-11 22:08:10,796::task::866::TaskManager.Task::(_setError) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool
>    spUUID, hostID, msdUUID, masterVersion, domainsMap)
>  File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool
>    res = pool.connect(hostID, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 625, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild
>    self.setMasterDomain(msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain
>    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
>StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'
>Thread-14::DEBUG::2014-04-11 22:08:10,797::task::885::TaskManager.Task::(_run) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::Task._run: 340cfcbe-3af8-4eb8-ab58-0f46c1e7601e ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:08:10,797::task::1211::TaskManager.Task::(stop) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:08:10,798::task::990::TaskManager.Task::(_decref) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:08:10,798::task::1168::TaskManager.Task::(prepare) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::aborting: Task is aborted: 'Cannot find master domain' - code 304
>Thread-14::DEBUG::2014-04-11 22:08:10,798::task::1173::TaskManager.Task::(prepare) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::Prepare: aborted: Cannot find master domain
>Thread-14::DEBUG::2014-04-11 22:08:10,799::task::990::TaskManager.Task::(_decref) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:08:10,799::task::925::TaskManager.Task::(_doAbort) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:08:10,799::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:08:10,800::task::595::TaskManager.Task::(_updateState) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:08:10,800::task::550::TaskManager.Task::(__state_aborting) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:08:10,800::task::595::TaskManager.Task::(_updateState) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:08:10,801::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:08:10,801::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:08:10,801::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}}
>Thread-14::DEBUG::2014-04-11 22:08:20,904::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6ead44fe]
>Thread-14::DEBUG::2014-04-11 22:08:20,907::task::595::TaskManager.Task::(_updateState) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:08:20,907::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:08:20,908::task::866::TaskManager.Task::(_setError) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:08:20,909::task::885::TaskManager.Task::(_run) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::Task._run: 070a06b5-72bb-4ce6-8480-7a56e606b17d ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:08:20,909::task::1211::TaskManager.Task::(stop) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:08:20,910::task::990::TaskManager.Task::(_decref) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:08:20,910::task::1168::TaskManager.Task::(prepare) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:08:20,910::task::1173::TaskManager.Task::(prepare) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:08:20,911::task::990::TaskManager.Task::(_decref) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:08:20,911::task::925::TaskManager.Task::(_doAbort) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:08:20,911::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:08:20,911::task::595::TaskManager.Task::(_updateState) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:08:20,912::task::550::TaskManager.Task::(__state_aborting) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:08:20,912::task::595::TaskManager.Task::(_updateState) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:08:20,912::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:08:20,913::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:08:20,913::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:08:20,917::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6ead44fe]
>Thread-14::DEBUG::2014-04-11 22:08:20,917::task::595::TaskManager.Task::(_updateState) Task=`15771a66-3978-4458-88d7-ac47a2424756`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:08:20,918::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-14::ERROR::2014-04-11 22:08:20,918::task::866::TaskManager.Task::(_setError) Task=`15771a66-3978-4458-88d7-ac47a2424756`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
>    raise se.SpmStatusError()
>SpmStatusError: Not SPM: ()
>Thread-14::DEBUG::2014-04-11 22:08:20,919::task::885::TaskManager.Task::(_run) Task=`15771a66-3978-4458-88d7-ac47a2424756`::Task._run: 15771a66-3978-4458-88d7-ac47a2424756 () {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:08:20,919::task::1211::TaskManager.Task::(stop) Task=`15771a66-3978-4458-88d7-ac47a2424756`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:08:20,919::task::990::TaskManager.Task::(_decref) Task=`15771a66-3978-4458-88d7-ac47a2424756`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:08:20,919::task::1168::TaskManager.Task::(prepare) Task=`15771a66-3978-4458-88d7-ac47a2424756`::aborting: Task is aborted: 'Not SPM' - code 654
>Thread-14::DEBUG::2014-04-11 22:08:20,920::task::1173::TaskManager.Task::(prepare) Task=`15771a66-3978-4458-88d7-ac47a2424756`::Prepare: aborted: Not SPM
>Thread-14::DEBUG::2014-04-11 22:08:20,920::task::990::TaskManager.Task::(_decref) Task=`15771a66-3978-4458-88d7-ac47a2424756`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:08:20,920::task::925::TaskManager.Task::(_doAbort) Task=`15771a66-3978-4458-88d7-ac47a2424756`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:08:20,921::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:08:20,921::task::595::TaskManager.Task::(_updateState) Task=`15771a66-3978-4458-88d7-ac47a2424756`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:08:20,921::task::550::TaskManager.Task::(__state_aborting) Task=`15771a66-3978-4458-88d7-ac47a2424756`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:08:20,921::task::595::TaskManager.Task::(_updateState) Task=`15771a66-3978-4458-88d7-ac47a2424756`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:08:20,922::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:08:20,922::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:08:20,922::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
>Thread-14::DEBUG::2014-04-11 22:08:21,038::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6ead44fe]
>Thread-14::DEBUG::2014-04-11 22:08:21,039::task::595::TaskManager.Task::(_updateState) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:08:21,039::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:08:21,040::task::866::TaskManager.Task::(_setError) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:08:21,042::task::885::TaskManager.Task::(_run) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::Task._run: caf5b536-8710-4bfe-add0-55a59758cf5a ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:08:21,042::task::1211::TaskManager.Task::(stop) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:08:21,042::task::990::TaskManager.Task::(_decref) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:08:21,043::task::1168::TaskManager.Task::(prepare) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:08:21,043::task::1173::TaskManager.Task::(prepare) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:08:21,043::task::990::TaskManager.Task::(_decref) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:08:21,044::task::925::TaskManager.Task::(_doAbort) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:08:21,044::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:08:21,044::task::595::TaskManager.Task::(_updateState) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:08:21,044::task::550::TaskManager.Task::(__state_aborting) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:08:21,045::task::595::TaskManager.Task::(_updateState) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:08:21,045::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:08:21,045::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:08:21,046::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:08:21,090::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6ead44fe]
>Thread-14::DEBUG::2014-04-11 22:08:21,091::task::595::TaskManager.Task::(_updateState) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:08:21,091::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:08:21,094::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f1375f11-a4c2-4e61-92d7-9e812132ebb6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:08:21,094::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:08:21,095::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:08:21,095::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f1375f11-a4c2-4e61-92d7-9e812132ebb6`::Granted request
>Thread-14::DEBUG::2014-04-11 22:08:21,096::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`3406dec5-f6c1-44f4-b2a3-d576719a6ddb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:08:21,096::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:08:21,097::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:08:21,097::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`3406dec5-f6c1-44f4-b2a3-d576719a6ddb`::Granted request
>Thread-14::INFO::2014-04-11 22:08:21,098::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c0b7170
>Thread-14::INFO::2014-04-11 22:08:21,098::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36)
>Thread-14::DEBUG::2014-04-11 22:08:21,098::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,099::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,099::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,099::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,100::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,100::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,101::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:08:21,101::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:08:21,101::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 22:08:21,101::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:08:21,102::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 22:08:21,102::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:08:21,163::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:08:21,164::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:08:21,286::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:08:21,287::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,287::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,288::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,288::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,289::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,289::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,289::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::ERROR::2014-04-11 22:08:21,290::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::ERROR::2014-04-11 22:08:21,290::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:08:21,290::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:08:21,294::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:08:21,635::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:08:21,646::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-14::DEBUG::2014-04-11 22:08:21,647::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:08:59,502::hsm::2387::Storage.HSM::(connectStorageServer) prefetch failed: {'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': <function findDomain at 0x1b75230>, '21619c8e-99ea-4813-be02-d708971e5393': <function findDomain at 0x1b75230>}
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2384, in connectStorageServer
>    doms = self.__prefetchDomains(domType, conObj)
>  File "/usr/share/vdsm/storage/hsm.py", line 2321, in __prefetchDomains
>    goop.glob.glob(os.path.join(lPath, uuidPatern)))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-28::DEBUG::2014-04-11 22:08:59,503::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain}
>Thread-28::INFO::2014-04-11 22:08:59,504::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}]}
>Thread-28::DEBUG::2014-04-11 22:08:59,516::task::1185::TaskManager.Task::(prepare) Task=`df78d592-6a16-4082-9277-2a886613ef82`::finished: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}]}
>Thread-28::DEBUG::2014-04-11 22:08:59,517::task::595::TaskManager.Task::(_updateState) Task=`df78d592-6a16-4082-9277-2a886613ef82`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:08:59,519::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:08:59,520::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:08:59,522::task::990::TaskManager.Task::(_decref) Task=`df78d592-6a16-4082-9277-2a886613ef82`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:08:59,548::task::595::TaskManager.Task::(_updateState) Task=`1eed29ad-0607-4037-9d60-20db8b7863d5`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:08:59,551::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:08:59,552::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:08:59,554::task::1185::TaskManager.Task::(prepare) Task=`1eed29ad-0607-4037-9d60-20db8b7863d5`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:08:59,556::task::595::TaskManager.Task::(_updateState) Task=`1eed29ad-0607-4037-9d60-20db8b7863d5`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:08:59,557::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:08:59,559::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:08:59,561::task::990::TaskManager.Task::(_decref) Task=`1eed29ad-0607-4037-9d60-20db8b7863d5`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:09:14,923::task::595::TaskManager.Task::(_updateState) Task=`584ed49f-bacf-42df-8d52-eb7c41eb68cb`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:09:14,924::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:09:14,924::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:09:14,925::task::1185::TaskManager.Task::(prepare) Task=`584ed49f-bacf-42df-8d52-eb7c41eb68cb`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:09:14,925::task::595::TaskManager.Task::(_updateState) Task=`584ed49f-bacf-42df-8d52-eb7c41eb68cb`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:09:14,925::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:09:14,926::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:09:14,926::task::990::TaskManager.Task::(_decref) Task=`584ed49f-bacf-42df-8d52-eb7c41eb68cb`::ref 0 aborting False
>Thread-29::WARNING::2014-04-11 22:09:21,697::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-30::WARNING::2014-04-11 22:09:21,703::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-29::DEBUG::2014-04-11 22:09:21,708::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-31::WARNING::2014-04-11 22:09:21,706::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-30::DEBUG::2014-04-11 22:09:21,711::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-31::DEBUG::2014-04-11 22:09:21,715::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-14::ERROR::2014-04-11 22:09:21,717::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',)
>Thread-14::DEBUG::2014-04-11 22:09:21,718::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:09:21,718::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:09:21,719::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:09:21,719::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 22:09:21,720::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 22:09:21,720::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:09:21,720::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:09:21,721::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-14::ERROR::2014-04-11 22:09:21,721::task::866::TaskManager.Task::(_setError) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool
>    spUUID, hostID, msdUUID, masterVersion, domainsMap)
>  File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool
>    res = pool.connect(hostID, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 625, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild
>    self.setMasterDomain(msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain
>    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
>StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'
>Thread-14::DEBUG::2014-04-11 22:09:21,722::task::885::TaskManager.Task::(_run) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::Task._run: 09e8e71b-c838-4573-a0ef-59f3063aa1fc ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:09:21,722::task::1211::TaskManager.Task::(stop) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:09:21,723::task::990::TaskManager.Task::(_decref) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:09:21,723::task::1168::TaskManager.Task::(prepare) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::aborting: Task is aborted: 'Cannot find master domain' - code 304
>Thread-14::DEBUG::2014-04-11 22:09:21,724::task::1173::TaskManager.Task::(prepare) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::Prepare: aborted: Cannot find master domain
>Thread-14::DEBUG::2014-04-11 22:09:21,724::task::990::TaskManager.Task::(_decref) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:09:21,724::task::925::TaskManager.Task::(_doAbort) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:09:21,725::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:09:21,725::task::595::TaskManager.Task::(_updateState) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:09:21,725::task::550::TaskManager.Task::(__state_aborting) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:09:21,726::task::595::TaskManager.Task::(_updateState) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:09:21,726::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:09:21,726::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:09:21,727::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}}
>Thread-14::DEBUG::2014-04-11 22:09:21,887::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [7de167af]
>Thread-14::DEBUG::2014-04-11 22:09:21,888::task::595::TaskManager.Task::(_updateState) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:09:21,889::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:09:21,889::task::866::TaskManager.Task::(_setError) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:09:21,890::task::885::TaskManager.Task::(_run) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::Task._run: 8bcbb6aa-d71f-44cd-bdcb-47c07661154d ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:09:21,890::task::1211::TaskManager.Task::(stop) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:09:21,891::task::990::TaskManager.Task::(_decref) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:09:21,891::task::1168::TaskManager.Task::(prepare) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:09:21,891::task::1173::TaskManager.Task::(prepare) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:09:21,892::task::990::TaskManager.Task::(_decref) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:09:21,892::task::925::TaskManager.Task::(_doAbort) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:09:21,892::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:09:21,892::task::595::TaskManager.Task::(_updateState) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:09:21,893::task::550::TaskManager.Task::(__state_aborting) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:09:21,893::task::595::TaskManager.Task::(_updateState) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:09:21,893::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:09:21,894::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:09:21,894::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:09:21,942::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [7de167af]
>Thread-14::DEBUG::2014-04-11 22:09:21,942::task::595::TaskManager.Task::(_updateState) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:09:21,943::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:09:21,944::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4e400ada-cd83-4e80-b309-ff62c3f5ae9c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:09:21,944::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:09:21,944::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:09:21,945::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4e400ada-cd83-4e80-b309-ff62c3f5ae9c`::Granted request
>Thread-14::DEBUG::2014-04-11 22:09:21,946::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`e490f3fa-3f5b-49b7-ae55-87e7e63dd91e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:09:21,946::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:09:21,947::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:09:21,947::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`e490f3fa-3f5b-49b7-ae55-87e7e63dd91e`::Granted request
>Thread-14::INFO::2014-04-11 22:09:21,948::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c12f908
>Thread-14::INFO::2014-04-11 22:09:21,948::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36)
>Thread-14::DEBUG::2014-04-11 22:09:21,948::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:21,949::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:21,949::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:21,950::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:21,950::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:21,951::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:21,951::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:09:21,951::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:09:21,952::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 22:09:21,952::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:09:21,952::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 22:09:21,953::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:09:21,996::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:09:21,997::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:09:22,126::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:09:22,127::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:22,128::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:22,128::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:22,129::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:22,129::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:22,129::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:22,130::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::ERROR::2014-04-11 22:09:22,130::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::ERROR::2014-04-11 22:09:22,131::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:09:22,131::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:09:22,134::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:09:22,498::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:09:22,500::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-14::DEBUG::2014-04-11 22:09:22,501::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:09:30,241::task::595::TaskManager.Task::(_updateState) Task=`6dcc0826-8170-4b76-a0de-7918118dd922`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:09:30,241::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:09:30,242::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:09:30,242::task::1185::TaskManager.Task::(prepare) Task=`6dcc0826-8170-4b76-a0de-7918118dd922`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:09:30,242::task::595::TaskManager.Task::(_updateState) Task=`6dcc0826-8170-4b76-a0de-7918118dd922`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:09:30,243::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:09:30,243::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:09:30,243::task::990::TaskManager.Task::(_decref) Task=`6dcc0826-8170-4b76-a0de-7918118dd922`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:09:45,574::task::595::TaskManager.Task::(_updateState) Task=`42d87c0c-63f1-4161-81a4-2d53a2008b7a`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:09:45,575::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:09:45,575::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:09:45,576::task::1185::TaskManager.Task::(prepare) Task=`42d87c0c-63f1-4161-81a4-2d53a2008b7a`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:09:45,576::task::595::TaskManager.Task::(_updateState) Task=`42d87c0c-63f1-4161-81a4-2d53a2008b7a`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:09:45,576::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:09:45,576::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:09:45,577::task::990::TaskManager.Task::(_decref) Task=`42d87c0c-63f1-4161-81a4-2d53a2008b7a`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:10:00,897::task::595::TaskManager.Task::(_updateState) Task=`aa06387d-ee84-4265-bade-34daf29bdd59`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:10:00,898::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:10:00,898::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:10:00,898::task::1185::TaskManager.Task::(prepare) Task=`aa06387d-ee84-4265-bade-34daf29bdd59`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:10:00,899::task::595::TaskManager.Task::(_updateState) Task=`aa06387d-ee84-4265-bade-34daf29bdd59`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:10:00,899::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:10:00,899::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:10:00,900::task::990::TaskManager.Task::(_decref) Task=`aa06387d-ee84-4265-bade-34daf29bdd59`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:10:16,313::task::595::TaskManager.Task::(_updateState) Task=`2c0b755b-70ea-4d54-9aba-4b36bfef7db8`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:10:16,314::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:10:16,314::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:10:16,315::task::1185::TaskManager.Task::(prepare) Task=`2c0b755b-70ea-4d54-9aba-4b36bfef7db8`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:10:16,315::task::595::TaskManager.Task::(_updateState) Task=`2c0b755b-70ea-4d54-9aba-4b36bfef7db8`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:10:16,315::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:10:16,316::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:10:16,316::task::990::TaskManager.Task::(_decref) Task=`2c0b755b-70ea-4d54-9aba-4b36bfef7db8`::ref 0 aborting False
>Thread-35::WARNING::2014-04-11 22:10:22,567::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-36::WARNING::2014-04-11 22:10:22,573::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-35::DEBUG::2014-04-11 22:10:22,580::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-37::WARNING::2014-04-11 22:10:22,575::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-36::DEBUG::2014-04-11 22:10:22,583::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-37::DEBUG::2014-04-11 22:10:22,587::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-14::ERROR::2014-04-11 22:10:22,589::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',)
>Thread-14::DEBUG::2014-04-11 22:10:22,590::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:10:22,591::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:10:22,591::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:10:22,591::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 22:10:22,592::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 22:10:22,592::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:10:22,592::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:10:22,593::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-14::ERROR::2014-04-11 22:10:22,593::task::866::TaskManager.Task::(_setError) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool
>    spUUID, hostID, msdUUID, masterVersion, domainsMap)
>  File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool
>    res = pool.connect(hostID, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 625, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild
>    self.setMasterDomain(msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain
>    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
>StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'
>Thread-14::DEBUG::2014-04-11 22:10:22,594::task::885::TaskManager.Task::(_run) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::Task._run: e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:10:22,594::task::1211::TaskManager.Task::(stop) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:10:22,595::task::990::TaskManager.Task::(_decref) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:10:22,595::task::1168::TaskManager.Task::(prepare) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::aborting: Task is aborted: 'Cannot find master domain' - code 304
>Thread-14::DEBUG::2014-04-11 22:10:22,596::task::1173::TaskManager.Task::(prepare) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::Prepare: aborted: Cannot find master domain
>Thread-14::DEBUG::2014-04-11 22:10:22,596::task::990::TaskManager.Task::(_decref) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:10:22,596::task::925::TaskManager.Task::(_doAbort) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:10:22,597::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:10:22,597::task::595::TaskManager.Task::(_updateState) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:10:22,597::task::550::TaskManager.Task::(__state_aborting) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:10:22,597::task::595::TaskManager.Task::(_updateState) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:10:22,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:10:22,598::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:10:22,598::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}}
>Thread-14::DEBUG::2014-04-11 22:10:22,690::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4406499a]
>Thread-14::DEBUG::2014-04-11 22:10:22,691::task::595::TaskManager.Task::(_updateState) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:10:22,692::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:10:22,694::task::866::TaskManager.Task::(_setError) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:10:22,695::task::885::TaskManager.Task::(_run) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::Task._run: aceb131b-cbe1-412a-b2e1-4ea192323454 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:10:22,696::task::1211::TaskManager.Task::(stop) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:10:22,696::task::990::TaskManager.Task::(_decref) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:10:22,696::task::1168::TaskManager.Task::(prepare) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:10:22,697::task::1173::TaskManager.Task::(prepare) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:10:22,697::task::990::TaskManager.Task::(_decref) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:10:22,697::task::925::TaskManager.Task::(_doAbort) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:10:22,698::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:10:22,698::task::595::TaskManager.Task::(_updateState) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:10:22,698::task::550::TaskManager.Task::(__state_aborting) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:10:22,699::task::595::TaskManager.Task::(_updateState) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:10:22,699::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:10:22,699::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:10:22,700::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:10:22,704::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4406499a]
>Thread-14::DEBUG::2014-04-11 22:10:22,704::task::595::TaskManager.Task::(_updateState) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:10:22,705::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-14::ERROR::2014-04-11 22:10:22,705::task::866::TaskManager.Task::(_setError) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
>    raise se.SpmStatusError()
>SpmStatusError: Not SPM: ()
>Thread-14::DEBUG::2014-04-11 22:10:22,706::task::885::TaskManager.Task::(_run) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::Task._run: 689eb1db-cb02-49a5-8d58-bff01b09eb91 () {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:10:22,706::task::1211::TaskManager.Task::(stop) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:10:22,707::task::990::TaskManager.Task::(_decref) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:10:22,707::task::1168::TaskManager.Task::(prepare) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::aborting: Task is aborted: 'Not SPM' - code 654
>Thread-14::DEBUG::2014-04-11 22:10:22,707::task::1173::TaskManager.Task::(prepare) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::Prepare: aborted: Not SPM
>Thread-14::DEBUG::2014-04-11 22:10:22,707::task::990::TaskManager.Task::(_decref) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:10:22,708::task::925::TaskManager.Task::(_doAbort) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:10:22,708::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:10:22,708::task::595::TaskManager.Task::(_updateState) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:10:22,709::task::550::TaskManager.Task::(__state_aborting) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:10:22,709::task::595::TaskManager.Task::(_updateState) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:10:22,709::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:10:22,710::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:10:22,710::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
>Thread-14::DEBUG::2014-04-11 22:10:22,830::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4406499a]
>Thread-14::DEBUG::2014-04-11 22:10:22,832::task::595::TaskManager.Task::(_updateState) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:10:22,832::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:10:22,833::task::866::TaskManager.Task::(_setError) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:10:22,834::task::885::TaskManager.Task::(_run) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::Task._run: 73e565a2-87cf-4511-a466-fb70604bd4be ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:10:22,834::task::1211::TaskManager.Task::(stop) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:10:22,835::task::990::TaskManager.Task::(_decref) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:10:22,835::task::1168::TaskManager.Task::(prepare) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:10:22,836::task::1173::TaskManager.Task::(prepare) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:10:22,836::task::990::TaskManager.Task::(_decref) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:10:22,837::task::925::TaskManager.Task::(_doAbort) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:10:22,837::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:10:22,838::task::595::TaskManager.Task::(_updateState) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:10:22,838::task::550::TaskManager.Task::(__state_aborting) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:10:22,839::task::595::TaskManager.Task::(_updateState) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:10:22,839::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:10:22,840::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:10:22,840::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:10:22,884::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4406499a]
>Thread-14::DEBUG::2014-04-11 22:10:22,885::task::595::TaskManager.Task::(_updateState) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:10:22,885::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:10:22,886::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a51af771-9905-4e57-91c5-aba8fb1003a1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:10:22,887::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:10:22,887::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:10:22,888::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a51af771-9905-4e57-91c5-aba8fb1003a1`::Granted request
>Thread-14::DEBUG::2014-04-11 22:10:22,889::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`4eed6604-b4da-47d0-80e4-7f81722b2ec1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:10:22,889::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:10:22,890::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:10:22,890::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`4eed6604-b4da-47d0-80e4-7f81722b2ec1`::Granted request
>Thread-14::INFO::2014-04-11 22:10:22,891::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c0d8290
>Thread-14::INFO::2014-04-11 22:10:22,891::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36)
>Thread-14::DEBUG::2014-04-11 22:10:22,892::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:22,892::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:22,893::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:22,893::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:22,893::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:22,894::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:22,894::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:10:22,895::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:10:22,895::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 22:10:22,895::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:10:22,895::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 22:10:22,896::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:10:22,954::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:10:22,955::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:10:23,113::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:10:23,114::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:23,114::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:23,115::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:23,115::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:23,116::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:23,116::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:23,117::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::ERROR::2014-04-11 22:10:23,117::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::ERROR::2014-04-11 22:10:23,117::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:10:23,118::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:10:23,121::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:10:23,478::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:10:23,481::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-14::DEBUG::2014-04-11 22:10:23,482::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:10:31,638::task::595::TaskManager.Task::(_updateState) Task=`c4e33ca2-e5c7-402b-bce5-7c8183a2dcd0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:10:31,639::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:10:31,639::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:10:31,639::task::1185::TaskManager.Task::(prepare) Task=`c4e33ca2-e5c7-402b-bce5-7c8183a2dcd0`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:10:31,639::task::595::TaskManager.Task::(_updateState) Task=`c4e33ca2-e5c7-402b-bce5-7c8183a2dcd0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:10:31,640::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:10:31,640::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:10:31,640::task::990::TaskManager.Task::(_decref) Task=`c4e33ca2-e5c7-402b-bce5-7c8183a2dcd0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:10:46,963::task::595::TaskManager.Task::(_updateState) Task=`0b23b7b8-52a8-49ef-b84e-25e1099cb64f`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:10:46,974::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:10:46,974::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:10:46,975::task::1185::TaskManager.Task::(prepare) Task=`0b23b7b8-52a8-49ef-b84e-25e1099cb64f`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:10:46,975::task::595::TaskManager.Task::(_updateState) Task=`0b23b7b8-52a8-49ef-b84e-25e1099cb64f`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:10:46,975::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:10:46,976::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:10:46,976::task::990::TaskManager.Task::(_decref) Task=`0b23b7b8-52a8-49ef-b84e-25e1099cb64f`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:11:02,273::task::595::TaskManager.Task::(_updateState) Task=`29121fff-2b45-4703-87ab-9160368a3c4b`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:11:02,274::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:11:02,274::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:11:02,275::task::1185::TaskManager.Task::(prepare) Task=`29121fff-2b45-4703-87ab-9160368a3c4b`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:11:02,275::task::595::TaskManager.Task::(_updateState) Task=`29121fff-2b45-4703-87ab-9160368a3c4b`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:11:02,275::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:11:02,276::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:11:02,276::task::990::TaskManager.Task::(_decref) Task=`29121fff-2b45-4703-87ab-9160368a3c4b`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:11:17,599::task::595::TaskManager.Task::(_updateState) Task=`206e204a-cb49-4902-8707-fa8b5b87d8f5`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:11:17,599::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:11:17,600::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:11:17,600::task::1185::TaskManager.Task::(prepare) Task=`206e204a-cb49-4902-8707-fa8b5b87d8f5`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:11:17,600::task::595::TaskManager.Task::(_updateState) Task=`206e204a-cb49-4902-8707-fa8b5b87d8f5`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:11:17,601::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:11:17,601::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:11:17,601::task::990::TaskManager.Task::(_decref) Task=`206e204a-cb49-4902-8707-fa8b5b87d8f5`::ref 0 aborting False
>Thread-41::WARNING::2014-04-11 22:11:23,529::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-42::WARNING::2014-04-11 22:11:23,536::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-41::DEBUG::2014-04-11 22:11:23,549::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-43::WARNING::2014-04-11 22:11:23,540::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-42::DEBUG::2014-04-11 22:11:23,552::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-43::DEBUG::2014-04-11 22:11:23,557::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-14::ERROR::2014-04-11 22:11:23,560::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',)
>Thread-14::DEBUG::2014-04-11 22:11:23,561::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:11:23,562::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:11:23,562::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:11:23,563::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 22:11:23,563::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 22:11:23,564::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:11:23,564::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:11:23,564::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-14::ERROR::2014-04-11 22:11:23,565::task::866::TaskManager.Task::(_setError) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool
>    spUUID, hostID, msdUUID, masterVersion, domainsMap)
>  File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool
>    res = pool.connect(hostID, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 625, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild
>    self.setMasterDomain(msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain
>    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
>StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'
>Thread-14::DEBUG::2014-04-11 22:11:23,566::task::885::TaskManager.Task::(_run) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::Task._run: 3eb6aee2-6b1b-48b1-b8ee-120fcae16b90 ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:11:23,566::task::1211::TaskManager.Task::(stop) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:11:23,566::task::990::TaskManager.Task::(_decref) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:11:23,567::task::1168::TaskManager.Task::(prepare) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::aborting: Task is aborted: 'Cannot find master domain' - code 304
>Thread-14::DEBUG::2014-04-11 22:11:23,567::task::1173::TaskManager.Task::(prepare) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::Prepare: aborted: Cannot find master domain
>Thread-14::DEBUG::2014-04-11 22:11:23,567::task::990::TaskManager.Task::(_decref) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:11:23,568::task::925::TaskManager.Task::(_doAbort) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:11:23,568::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:11:23,568::task::595::TaskManager.Task::(_updateState) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:11:23,569::task::550::TaskManager.Task::(__state_aborting) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:11:23,569::task::595::TaskManager.Task::(_updateState) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:11:23,569::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:11:23,570::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:11:23,570::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}}
>Thread-28::DEBUG::2014-04-11 22:11:23,803::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [200912ce]
>Thread-28::DEBUG::2014-04-11 22:11:23,808::task::595::TaskManager.Task::(_updateState) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:11:23,809::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-28::ERROR::2014-04-11 22:11:23,809::task::866::TaskManager.Task::(_setError) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
>    raise se.SpmStatusError()
>SpmStatusError: Not SPM: ()
>Thread-28::DEBUG::2014-04-11 22:11:23,810::task::885::TaskManager.Task::(_run) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::Task._run: e28a0121-1ce3-4899-b539-ea0280711dbb () {} failed - stopping task
>Thread-28::DEBUG::2014-04-11 22:11:23,810::task::1211::TaskManager.Task::(stop) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::stopping in state preparing (force False)
>Thread-28::DEBUG::2014-04-11 22:11:23,810::task::990::TaskManager.Task::(_decref) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::ref 1 aborting True
>Thread-28::INFO::2014-04-11 22:11:23,811::task::1168::TaskManager.Task::(prepare) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::aborting: Task is aborted: 'Not SPM' - code 654
>Thread-28::DEBUG::2014-04-11 22:11:23,811::task::1173::TaskManager.Task::(prepare) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::Prepare: aborted: Not SPM
>Thread-28::DEBUG::2014-04-11 22:11:23,811::task::990::TaskManager.Task::(_decref) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::ref 0 aborting True
>Thread-28::DEBUG::2014-04-11 22:11:23,812::task::925::TaskManager.Task::(_doAbort) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::Task._doAbort: force False
>Thread-28::DEBUG::2014-04-11 22:11:23,812::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:11:23,812::task::595::TaskManager.Task::(_updateState) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::moving from state preparing -> state aborting
>Thread-28::DEBUG::2014-04-11 22:11:23,813::task::550::TaskManager.Task::(__state_aborting) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::_aborting: recover policy none
>Thread-28::DEBUG::2014-04-11 22:11:23,813::task::595::TaskManager.Task::(_updateState) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::moving from state aborting -> state failed
>Thread-28::DEBUG::2014-04-11 22:11:23,813::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:11:23,814::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::ERROR::2014-04-11 22:11:23,814::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
>Thread-28::DEBUG::2014-04-11 22:11:23,908::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [200912ce]
>Thread-28::DEBUG::2014-04-11 22:11:23,909::task::595::TaskManager.Task::(_updateState) Task=`7fa4df86-db61-49a0-8afe-1b35d290fb8b`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:11:23,909::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, remove=False, options=None)
>Thread-28::WARNING::2014-04-11 22:11:23,910::hsm::1116::Storage.HSM::(disconnectStoragePool) disconnect sp: a106ab81-9d5f-49c1-aeaf-832a137b708c failed. Known pools {}
>Thread-28::INFO::2014-04-11 22:11:23,910::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None
>Thread-28::DEBUG::2014-04-11 22:11:23,910::task::1185::TaskManager.Task::(prepare) Task=`7fa4df86-db61-49a0-8afe-1b35d290fb8b`::finished: None
>Thread-28::DEBUG::2014-04-11 22:11:23,911::task::595::TaskManager.Task::(_updateState) Task=`7fa4df86-db61-49a0-8afe-1b35d290fb8b`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:11:23,911::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:11:23,911::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:11:23,912::task::990::TaskManager.Task::(_decref) Task=`7fa4df86-db61-49a0-8afe-1b35d290fb8b`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:11:23,956::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [200912ce]
>Thread-28::DEBUG::2014-04-11 22:11:23,957::task::595::TaskManager.Task::(_updateState) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:11:23,958::logUtils::44::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', poolName='SuprTEK', masterDom='3406665e-4adc-4fd4-aa1e-037547b29adb', domDict={'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'active', '21619c8e-99ea-4813-be02-d708971e5393': 'active'}, masterVersion=37, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=1, options=None)
>Thread-28::INFO::2014-04-11 22:11:23,958::hsm::1852::Storage.HSM::(reconstructMaster) spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c master=3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-28::INFO::2014-04-11 22:11:23,958::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e9c125830
>Thread-28::ERROR::2014-04-11 22:11:23,959::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-28::ERROR::2014-04-11 22:11:23,959::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-28::DEBUG::2014-04-11 22:11:23,960::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:11:23,961::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:11:24,325::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-28::WARNING::2014-04-11 22:11:24,330::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-28::DEBUG::2014-04-11 22:11:24,331::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:11:30,400::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [2e43b400]
>Thread-14::DEBUG::2014-04-11 22:11:30,402::task::595::TaskManager.Task::(_updateState) Task=`b8db45e8-e128-44b3-8f29-1914681f9624`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:11:30,402::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'ashtistg01.suprtekstic.com:/storage/infrastructure', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e', 'port': ''}], options=None)
>Thread-14::DEBUG::2014-04-11 22:11:30,415::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/infrastructure /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure' (cwd None)
>Thread-14::ERROR::2014-04-11 22:12:22,787::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-14::ERROR::2014-04-11 22:12:22,792::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-14::DEBUG::2014-04-11 22:12:22,794::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain}
>Thread-14::INFO::2014-04-11 22:12:22,796::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}]}
>Thread-14::DEBUG::2014-04-11 22:12:22,797::task::1185::TaskManager.Task::(prepare) Task=`b8db45e8-e128-44b3-8f29-1914681f9624`::finished: {'statuslist': [{'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}]}
>Thread-14::DEBUG::2014-04-11 22:12:22,797::task::595::TaskManager.Task::(_updateState) Task=`b8db45e8-e128-44b3-8f29-1914681f9624`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:12:22,799::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:12:22,800::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:12:22,801::task::990::TaskManager.Task::(_decref) Task=`b8db45e8-e128-44b3-8f29-1914681f9624`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:12:22,845::task::595::TaskManager.Task::(_updateState) Task=`973611d7-8fe8-4801-9ad8-ec9776b694fd`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:12:22,853::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:12:22,860::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-14::DEBUG::2014-04-11 22:12:22,865::task::1185::TaskManager.Task::(prepare) Task=`973611d7-8fe8-4801-9ad8-ec9776b694fd`::finished: {}
>Thread-14::DEBUG::2014-04-11 22:12:22,872::task::595::TaskManager.Task::(_updateState) Task=`973611d7-8fe8-4801-9ad8-ec9776b694fd`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:12:22,876::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:12:22,880::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:12:22,886::task::990::TaskManager.Task::(_decref) Task=`973611d7-8fe8-4801-9ad8-ec9776b694fd`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:12:22,923::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6d750a9d]
>Thread-14::DEBUG::2014-04-11 22:12:22,925::task::595::TaskManager.Task::(_updateState) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:12:22,926::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:12:22,927::task::866::TaskManager.Task::(_setError) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:12:22,929::task::885::TaskManager.Task::(_run) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::Task._run: 08c5c6da-1f8d-4add-bbee-f42ac0831116 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:12:22,930::task::1211::TaskManager.Task::(stop) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:12:22,931::task::990::TaskManager.Task::(_decref) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:12:22,932::task::1168::TaskManager.Task::(prepare) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:12:22,933::task::1173::TaskManager.Task::(prepare) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:12:22,934::task::990::TaskManager.Task::(_decref) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:12:22,934::task::925::TaskManager.Task::(_doAbort) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:12:22,935::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:12:22,936::task::595::TaskManager.Task::(_updateState) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:12:22,937::task::550::TaskManager.Task::(__state_aborting) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:12:22,937::task::595::TaskManager.Task::(_updateState) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:12:22,938::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:12:22,940::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:12:22,941::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:12:23,033::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6d750a9d]
>Thread-14::DEBUG::2014-04-11 22:12:23,035::task::595::TaskManager.Task::(_updateState) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:12:23,035::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='3406665e-4adc-4fd4-aa1e-037547b29adb', masterVersion=37, domainsMap=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:12:23,039::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e9c3b498-50aa-4458-8c92-0b52e1d9e92d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:12:23,039::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:12:23,041::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:12:23,041::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e9c3b498-50aa-4458-8c92-0b52e1d9e92d`::Granted request
>Thread-14::DEBUG::2014-04-11 22:12:23,045::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`3e1d779c-fad3-412d-bc1c-9eddde0d9126`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:12:23,046::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:12:23,047::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:12:23,048::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`3e1d779c-fad3-412d-bc1c-9eddde0d9126`::Granted request
>Thread-14::INFO::2014-04-11 22:12:23,050::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c075368
>Thread-14::INFO::2014-04-11 22:12:23,050::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: 3406665e-4adc-4fd4-aa1e-037547b29adb (ver = 37)
>Thread-14::DEBUG::2014-04-11 22:12:23,052::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:23,053::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:23,055::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:23,056::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:23,057::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:23,059::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-47::WARNING::2014-04-11 22:12:24,387::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-47::DEBUG::2014-04-11 22:12:24,399::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-48::WARNING::2014-04-11 22:12:24,399::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-49::WARNING::2014-04-11 22:12:24,402::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-48::DEBUG::2014-04-11 22:12:24,408::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-49::DEBUG::2014-04-11 22:12:24,409::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-28::ERROR::2014-04-11 22:12:24,412::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:12:24,413::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:12:24,415::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-28::ERROR::2014-04-11 22:12:24,413::task::866::TaskManager.Task::(_setError) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1862, in reconstructMaster
>    self.validateSdUUID(masterDom)
>  File "/usr/share/vdsm/storage/hsm.py", line 270, in validateSdUUID
>    sdDom = sdCache.produce(sdUUID=sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:12:24,417::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-28::DEBUG::2014-04-11 22:12:24,418::task::885::TaskManager.Task::(_run) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::Task._run: f4e11904-c42d-46f9-b869-dfcf9616fca1 ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 'SuprTEK', '3406665e-4adc-4fd4-aa1e-037547b29adb', {'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'active', '21619c8e-99ea-4813-be02-d708971e5393': 'active'}, 37, None, 5, 60, 10, 3, 1) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:12:24,419::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-28::DEBUG::2014-04-11 22:12:24,421::task::1211::TaskManager.Task::(stop) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:12:24,422::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-28::DEBUG::2014-04-11 22:12:24,424::task::990::TaskManager.Task::(_decref) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::ref 1 aborting True
>Thread-14::DEBUG::2014-04-11 22:12:24,425::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-28::INFO::2014-04-11 22:12:24,426::task::1168::TaskManager.Task::(prepare) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
>Thread-28::DEBUG::2014-04-11 22:12:24,437::task::1173::TaskManager.Task::(prepare) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::Prepare: aborted: Storage domain does not exist
>Thread-28::DEBUG::2014-04-11 22:12:24,438::task::990::TaskManager.Task::(_decref) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::ref 0 aborting True
>Thread-28::DEBUG::2014-04-11 22:12:24,438::task::925::TaskManager.Task::(_doAbort) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::Task._doAbort: force False
>Thread-28::DEBUG::2014-04-11 22:12:24,439::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:12:24,439::task::595::TaskManager.Task::(_updateState) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::moving from state preparing -> state aborting
>Thread-28::DEBUG::2014-04-11 22:12:24,440::task::550::TaskManager.Task::(__state_aborting) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::_aborting: recover policy none
>Thread-28::DEBUG::2014-04-11 22:12:24,440::task::595::TaskManager.Task::(_updateState) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::moving from state aborting -> state failed
>Thread-28::DEBUG::2014-04-11 22:12:24,441::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:12:24,441::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::ERROR::2014-04-11 22:12:24,442::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)", 'code': 358}}
>Thread-14::DEBUG::2014-04-11 22:12:24,458::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:12:24,458::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:12:24,612::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:12:24,613::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:24,614::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:24,614::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:24,615::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:24,615::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:24,616::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:24,616::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::ERROR::2014-04-11 22:12:24,616::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-14::ERROR::2014-04-11 22:12:24,617::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-14::DEBUG::2014-04-11 22:12:24,617::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:12:24,620::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:12:24,991::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:12:24,993::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-14::DEBUG::2014-04-11 22:12:24,993::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:12:38,278::task::595::TaskManager.Task::(_updateState) Task=`236b548a-14d8-4b8d-b9ea-aebe54b6adc3`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:12:38,280::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:12:38,280::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:12:38,280::task::1185::TaskManager.Task::(prepare) Task=`236b548a-14d8-4b8d-b9ea-aebe54b6adc3`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:12:38,281::task::595::TaskManager.Task::(_updateState) Task=`236b548a-14d8-4b8d-b9ea-aebe54b6adc3`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:12:38,281::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:12:38,281::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:12:38,282::task::990::TaskManager.Task::(_decref) Task=`236b548a-14d8-4b8d-b9ea-aebe54b6adc3`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:12:53,589::task::595::TaskManager.Task::(_updateState) Task=`60bcdbbf-1636-4ef3-8537-c6998b6421e1`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:12:53,590::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:12:53,590::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:12:53,590::task::1185::TaskManager.Task::(prepare) Task=`60bcdbbf-1636-4ef3-8537-c6998b6421e1`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:12:53,591::task::595::TaskManager.Task::(_updateState) Task=`60bcdbbf-1636-4ef3-8537-c6998b6421e1`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:12:53,591::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:12:53,591::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:12:53,592::task::990::TaskManager.Task::(_decref) Task=`60bcdbbf-1636-4ef3-8537-c6998b6421e1`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:13:08,901::task::595::TaskManager.Task::(_updateState) Task=`bc5b9679-c0ca-490b-8a2d-b1be06ca5bd0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:13:08,901::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:13:08,902::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:13:08,902::task::1185::TaskManager.Task::(prepare) Task=`bc5b9679-c0ca-490b-8a2d-b1be06ca5bd0`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:13:08,902::task::595::TaskManager.Task::(_updateState) Task=`bc5b9679-c0ca-490b-8a2d-b1be06ca5bd0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:13:08,903::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:13:08,903::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:13:08,903::task::990::TaskManager.Task::(_decref) Task=`bc5b9679-c0ca-490b-8a2d-b1be06ca5bd0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:13:24,233::task::595::TaskManager.Task::(_updateState) Task=`4071e668-41a8-4d03-b074-87c8d51e6d80`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:13:24,236::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-28::INFO::2014-04-11 22:13:24,238::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
>Thread-28::DEBUG::2014-04-11 22:13:24,239::task::1185::TaskManager.Task::(prepare) Task=`4071e668-41a8-4d03-b074-87c8d51e6d80`::finished: {}
>Thread-28::DEBUG::2014-04-11 22:13:24,239::task::595::TaskManager.Task::(_updateState) Task=`4071e668-41a8-4d03-b074-87c8d51e6d80`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:13:24,240::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:13:24,241::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:13:24,243::task::990::TaskManager.Task::(_decref) Task=`4071e668-41a8-4d03-b074-87c8d51e6d80`::ref 0 aborting False
>Thread-54::WARNING::2014-04-11 22:13:25,049::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-55::WARNING::2014-04-11 22:13:25,051::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-54::DEBUG::2014-04-11 22:13:25,064::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-55::DEBUG::2014-04-11 22:13:25,067::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-14::ERROR::2014-04-11 22:13:25,070::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:13:25,071::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:13:25,071::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:13:25,071::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:13:25,072::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 22:13:25,072::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 22:13:25,073::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:13:25,073::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:13:25,073::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-14::ERROR::2014-04-11 22:13:25,074::task::866::TaskManager.Task::(_setError) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool
>    spUUID, hostID, msdUUID, masterVersion, domainsMap)
>  File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool
>    res = pool.connect(hostID, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 625, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild
>    self.setMasterDomain(msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain
>    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
>StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=3406665e-4adc-4fd4-aa1e-037547b29adb'
>Thread-14::DEBUG::2014-04-11 22:13:25,075::task::885::TaskManager.Task::(_run) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::Task._run: f4279d64-2d14-44e0-a81c-131804e3f4f7 ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, '3406665e-4adc-4fd4-aa1e-037547b29adb', 37, None) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:13:25,075::task::1211::TaskManager.Task::(stop) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:13:25,076::task::990::TaskManager.Task::(_decref) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:13:25,076::task::1168::TaskManager.Task::(prepare) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::aborting: Task is aborted: 'Cannot find master domain' - code 304
>Thread-14::DEBUG::2014-04-11 22:13:25,076::task::1173::TaskManager.Task::(prepare) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::Prepare: aborted: Cannot find master domain
>Thread-14::DEBUG::2014-04-11 22:13:25,077::task::990::TaskManager.Task::(_decref) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:13:25,077::task::925::TaskManager.Task::(_doAbort) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:13:25,077::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:13:25,078::task::595::TaskManager.Task::(_updateState) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:13:25,078::task::550::TaskManager.Task::(__state_aborting) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:13:25,078::task::595::TaskManager.Task::(_updateState) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:13:25,079::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:13:25,079::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:13:25,079::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=3406665e-4adc-4fd4-aa1e-037547b29adb'", 'code': 304}}
>Thread-14::DEBUG::2014-04-11 22:13:25,282::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [29eaa508]
>Thread-14::DEBUG::2014-04-11 22:13:25,283::task::595::TaskManager.Task::(_updateState) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:13:25,284::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:13:25,284::task::866::TaskManager.Task::(_setError) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-14::DEBUG::2014-04-11 22:13:25,285::task::885::TaskManager.Task::(_run) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::Task._run: 61093c90-ac6a-475d-98a0-c14276df11e1 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:13:25,285::task::1211::TaskManager.Task::(stop) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:13:25,285::task::990::TaskManager.Task::(_decref) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:13:25,286::task::1168::TaskManager.Task::(prepare) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-14::DEBUG::2014-04-11 22:13:25,286::task::1173::TaskManager.Task::(prepare) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-14::DEBUG::2014-04-11 22:13:25,286::task::990::TaskManager.Task::(_decref) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:13:25,287::task::925::TaskManager.Task::(_doAbort) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:13:25,287::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:13:25,287::task::595::TaskManager.Task::(_updateState) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:13:25,288::task::550::TaskManager.Task::(__state_aborting) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:13:25,288::task::595::TaskManager.Task::(_updateState) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:13:25,288::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:13:25,289::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:13:25,289::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-14::DEBUG::2014-04-11 22:13:25,336::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [29eaa508]
>Thread-14::DEBUG::2014-04-11 22:13:25,337::task::595::TaskManager.Task::(_updateState) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:13:25,338::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=37, domainsMap=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:13:25,338::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`3e8aafa4-01c2-4301-aa64-5c10076bf1c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:13:25,339::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:13:25,339::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:13:25,340::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`3e8aafa4-01c2-4301-aa64-5c10076bf1c2`::Granted request
>Thread-14::DEBUG::2014-04-11 22:13:25,341::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`7ffc5d06-6de3-4202-a871-44660affb354`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:13:25,341::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:13:25,341::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:13:25,342::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`7ffc5d06-6de3-4202-a871-44660affb354`::Granted request
>Thread-14::INFO::2014-04-11 22:13:25,343::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e840890e0
>Thread-14::INFO::2014-04-11 22:13:25,343::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 37)
>Thread-14::DEBUG::2014-04-11 22:13:25,343::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,344::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,344::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,345::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,345::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,346::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,346::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:13:25,346::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:13:25,347::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 22:13:25,347::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:13:25,347::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 22:13:25,348::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:13:25,393::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:13:25,394::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:13:25,534::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:13:25,535::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,535::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,536::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,536::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,537::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,537::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,537::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::ERROR::2014-04-11 22:13:25,538::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::ERROR::2014-04-11 22:13:25,538::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:13:25,539::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,542::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:13:25,892::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:13:25,894::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-14::DEBUG::2014-04-11 22:13:25,894::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:25,972::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:13:26,246::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-14::DEBUG::2014-04-11 22:13:26,264::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=36', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a76d9a8af2ad818a809eee527dd4e251532a84c3']
>Thread-14::DEBUG::2014-04-11 22:13:26,267::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-14::DEBUG::2014-04-11 22:13:26,268::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS'
>Thread-14::DEBUG::2014-04-11 22:13:26,269::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS'
>Thread-14::ERROR::2014-04-11 22:13:26,270::spbackends::282::Storage.StoragePoolDiskBackend::(validateMasterDomainVersion) Requested master domain a033c2ac-0d01-490c-9552-99ca53d6a64a does not have expected version 37 it is version 36
>Thread-14::DEBUG::2014-04-11 22:13:26,270::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:13:26,271::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:13:26,271::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:13:26,271::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 22:13:26,272::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 22:13:26,272::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:13:26,273::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:13:26,273::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-14::ERROR::2014-04-11 22:13:26,273::task::866::TaskManager.Task::(_setError) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool
>    spUUID, hostID, msdUUID, masterVersion, domainsMap)
>  File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool
>    res = pool.connect(hostID, msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 625, in connect
>    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild
>    self.setMasterDomain(msdUUID, masterVersion)
>  File "/usr/share/vdsm/storage/sp.py", line 1364, in setMasterDomain
>    self._backend.validateMasterDomainVersion(domain, masterVersion)
>  File "/usr/share/vdsm/storage/spbackends.py", line 283, in validateMasterDomainVersion
>    raise se.StoragePoolWrongMaster(self.spUUID, masterDomain.sdUUID)
>StoragePoolWrongMaster: Wrong Master domain or its version: 'SD=a033c2ac-0d01-490c-9552-99ca53d6a64a, pool=a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:13:26,283::task::885::TaskManager.Task::(_run) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::Task._run: feb8d088-f530-4b72-8cb1-f46ab75a6ddf ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 37, None) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:13:26,283::task::1211::TaskManager.Task::(stop) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:13:26,283::task::990::TaskManager.Task::(_decref) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:13:26,284::task::1168::TaskManager.Task::(prepare) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::aborting: Task is aborted: 'Wrong Master domain or its version' - code 324
>Thread-14::DEBUG::2014-04-11 22:13:26,284::task::1173::TaskManager.Task::(prepare) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::Prepare: aborted: Wrong Master domain or its version
>Thread-14::DEBUG::2014-04-11 22:13:26,285::task::990::TaskManager.Task::(_decref) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:13:26,285::task::925::TaskManager.Task::(_doAbort) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:13:26,285::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:13:26,286::task::595::TaskManager.Task::(_updateState) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:13:26,286::task::550::TaskManager.Task::(__state_aborting) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:13:26,286::task::595::TaskManager.Task::(_updateState) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:13:26,287::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:13:26,287::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:13:26,287::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Wrong Master domain or its version: 'SD=a033c2ac-0d01-490c-9552-99ca53d6a64a, pool=a106ab81-9d5f-49c1-aeaf-832a137b708c'", 'code': 324}}
>Thread-14::DEBUG::2014-04-11 22:13:26,437::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [5f90a775]
>Thread-14::DEBUG::2014-04-11 22:13:26,438::task::595::TaskManager.Task::(_updateState) Task=`79f16959-34cd-4c38-bd5b-07fc4634baba`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:13:26,439::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, remove=False, options=None)
>Thread-14::WARNING::2014-04-11 22:13:26,439::hsm::1116::Storage.HSM::(disconnectStoragePool) disconnect sp: a106ab81-9d5f-49c1-aeaf-832a137b708c failed. Known pools {}
>Thread-14::INFO::2014-04-11 22:13:26,439::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None
>Thread-14::DEBUG::2014-04-11 22:13:26,440::task::1185::TaskManager.Task::(prepare) Task=`79f16959-34cd-4c38-bd5b-07fc4634baba`::finished: None
>Thread-14::DEBUG::2014-04-11 22:13:26,440::task::595::TaskManager.Task::(_updateState) Task=`79f16959-34cd-4c38-bd5b-07fc4634baba`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:13:26,440::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:13:26,441::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:13:26,441::task::990::TaskManager.Task::(_decref) Task=`79f16959-34cd-4c38-bd5b-07fc4634baba`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:13:26,484::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [5f90a775]
>Thread-14::DEBUG::2014-04-11 22:13:26,485::task::595::TaskManager.Task::(_updateState) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:13:26,486::logUtils::44::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', poolName='SuprTEK', masterDom='a033c2ac-0d01-490c-9552-99ca53d6a64a', domDict={'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'active', '21619c8e-99ea-4813-be02-d708971e5393': 'active'}, masterVersion=38, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=1, options=None)
>Thread-14::INFO::2014-04-11 22:13:26,486::hsm::1852::Storage.HSM::(reconstructMaster) spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c master=a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::INFO::2014-04-11 22:13:26,487::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e9c110dd0
>Thread-14::INFO::2014-04-11 22:13:26,487::fileSD::316::Storage.StorageDomain::(validate) sdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:13:26,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=36', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a76d9a8af2ad818a809eee527dd4e251532a84c3']
>Thread-14::DEBUG::2014-04-11 22:13:26,505::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`cb7b76ab-6035-405f-bb8a-721329082378`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1867' at 'reconstructMaster'
>Thread-14::DEBUG::2014-04-11 22:13:26,506::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:13:26,506::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:13:26,506::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`cb7b76ab-6035-405f-bb8a-721329082378`::Granted request
>Thread-14::DEBUG::2014-04-11 22:13:26,507::task::827::TaskManager.Task::(resourceAcquired) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive)
>Thread-14::DEBUG::2014-04-11 22:13:26,508::task::990::TaskManager.Task::(_decref) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::ref 1 aborting False
>Thread-14::INFO::2014-04-11 22:13:26,508::sp::679::Storage.StoragePool::(reconstructMaster) spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c hostId=1 poolName=SuprTEK msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a domDict={'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'active', '21619c8e-99ea-4813-be02-d708971e5393': 'active'} masterVersion=38 leaseparams=({'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5})
>Thread-14::INFO::2014-04-11 22:13:26,508::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-28::DEBUG::2014-04-11 22:13:35,210::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:13:35,211::task::595::TaskManager.Task::(_updateState) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:13:35,212::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-28::ERROR::2014-04-11 22:13:35,212::task::866::TaskManager.Task::(_setError) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-28::DEBUG::2014-04-11 22:13:35,213::task::885::TaskManager.Task::(_run) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::Task._run: 7c9bf435-18ec-4e04-b2a2-9032d70a5669 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-28::DEBUG::2014-04-11 22:13:35,214::task::1211::TaskManager.Task::(stop) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::stopping in state preparing (force False)
>Thread-28::DEBUG::2014-04-11 22:13:35,214::task::990::TaskManager.Task::(_decref) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::ref 1 aborting True
>Thread-28::INFO::2014-04-11 22:13:35,214::task::1168::TaskManager.Task::(prepare) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-28::DEBUG::2014-04-11 22:13:35,214::task::1173::TaskManager.Task::(prepare) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-28::DEBUG::2014-04-11 22:13:35,215::task::990::TaskManager.Task::(_decref) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::ref 0 aborting True
>Thread-28::DEBUG::2014-04-11 22:13:35,215::task::925::TaskManager.Task::(_doAbort) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::Task._doAbort: force False
>Thread-28::DEBUG::2014-04-11 22:13:35,215::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:13:35,216::task::595::TaskManager.Task::(_updateState) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::moving from state preparing -> state aborting
>Thread-28::DEBUG::2014-04-11 22:13:35,216::task::550::TaskManager.Task::(__state_aborting) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::_aborting: recover policy none
>Thread-28::DEBUG::2014-04-11 22:13:35,216::task::595::TaskManager.Task::(_updateState) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::moving from state aborting -> state failed
>Thread-28::DEBUG::2014-04-11 22:13:35,216::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:13:35,217::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::ERROR::2014-04-11 22:13:35,217::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-28::DEBUG::2014-04-11 22:13:35,221::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:13:35,222::task::595::TaskManager.Task::(_updateState) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:13:35,222::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-28::ERROR::2014-04-11 22:13:35,222::task::866::TaskManager.Task::(_setError) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
>    raise se.SpmStatusError()
>SpmStatusError: Not SPM: ()
>Thread-28::DEBUG::2014-04-11 22:13:35,223::task::885::TaskManager.Task::(_run) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::Task._run: ddcfbb3b-268f-4fff-ae92-a8a40beef2e2 () {} failed - stopping task
>Thread-28::DEBUG::2014-04-11 22:13:35,223::task::1211::TaskManager.Task::(stop) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::stopping in state preparing (force False)
>Thread-28::DEBUG::2014-04-11 22:13:35,224::task::990::TaskManager.Task::(_decref) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::ref 1 aborting True
>Thread-28::INFO::2014-04-11 22:13:35,224::task::1168::TaskManager.Task::(prepare) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::aborting: Task is aborted: 'Not SPM' - code 654
>Thread-28::DEBUG::2014-04-11 22:13:35,224::task::1173::TaskManager.Task::(prepare) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::Prepare: aborted: Not SPM
>Thread-28::DEBUG::2014-04-11 22:13:35,224::task::990::TaskManager.Task::(_decref) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::ref 0 aborting True
>Thread-28::DEBUG::2014-04-11 22:13:35,225::task::925::TaskManager.Task::(_doAbort) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::Task._doAbort: force False
>Thread-28::DEBUG::2014-04-11 22:13:35,225::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:13:35,225::task::595::TaskManager.Task::(_updateState) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::moving from state preparing -> state aborting
>Thread-28::DEBUG::2014-04-11 22:13:35,226::task::550::TaskManager.Task::(__state_aborting) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::_aborting: recover policy none
>Thread-28::DEBUG::2014-04-11 22:13:35,226::task::595::TaskManager.Task::(_updateState) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::moving from state aborting -> state failed
>Thread-28::DEBUG::2014-04-11 22:13:35,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:13:35,226::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::ERROR::2014-04-11 22:13:35,227::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
>Thread-28::DEBUG::2014-04-11 22:13:35,343::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:13:35,344::task::595::TaskManager.Task::(_updateState) Task=`be16651a-0402-420a-b0d6-08f782484585`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:13:35,344::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-28::ERROR::2014-04-11 22:13:35,345::task::866::TaskManager.Task::(_setError) Task=`be16651a-0402-420a-b0d6-08f782484585`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus
>    pool = self.getPool(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool
>    raise se.StoragePoolUnknown(spUUID)
>StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-28::DEBUG::2014-04-11 22:13:35,345::task::885::TaskManager.Task::(_run) Task=`be16651a-0402-420a-b0d6-08f782484585`::Task._run: be16651a-0402-420a-b0d6-08f782484585 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-28::DEBUG::2014-04-11 22:13:35,346::task::1211::TaskManager.Task::(stop) Task=`be16651a-0402-420a-b0d6-08f782484585`::stopping in state preparing (force False)
>Thread-28::DEBUG::2014-04-11 22:13:35,346::task::990::TaskManager.Task::(_decref) Task=`be16651a-0402-420a-b0d6-08f782484585`::ref 1 aborting True
>Thread-28::INFO::2014-04-11 22:13:35,346::task::1168::TaskManager.Task::(prepare) Task=`be16651a-0402-420a-b0d6-08f782484585`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
>Thread-28::DEBUG::2014-04-11 22:13:35,347::task::1173::TaskManager.Task::(prepare) Task=`be16651a-0402-420a-b0d6-08f782484585`::Prepare: aborted: Unknown pool id, pool not connected
>Thread-28::DEBUG::2014-04-11 22:13:35,347::task::990::TaskManager.Task::(_decref) Task=`be16651a-0402-420a-b0d6-08f782484585`::ref 0 aborting True
>Thread-28::DEBUG::2014-04-11 22:13:35,347::task::925::TaskManager.Task::(_doAbort) Task=`be16651a-0402-420a-b0d6-08f782484585`::Task._doAbort: force False
>Thread-28::DEBUG::2014-04-11 22:13:35,348::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:13:35,348::task::595::TaskManager.Task::(_updateState) Task=`be16651a-0402-420a-b0d6-08f782484585`::moving from state preparing -> state aborting
>Thread-28::DEBUG::2014-04-11 22:13:35,348::task::550::TaskManager.Task::(__state_aborting) Task=`be16651a-0402-420a-b0d6-08f782484585`::_aborting: recover policy none
>Thread-28::DEBUG::2014-04-11 22:13:35,349::task::595::TaskManager.Task::(_updateState) Task=`be16651a-0402-420a-b0d6-08f782484585`::moving from state aborting -> state failed
>Thread-28::DEBUG::2014-04-11 22:13:35,349::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:13:35,349::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::ERROR::2014-04-11 22:13:35,350::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}}
>Thread-28::DEBUG::2014-04-11 22:13:35,393::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:13:35,394::task::595::TaskManager.Task::(_updateState) Task=`bb2d61f2-9971-4bcc-9374-5e2746c7e69f`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:13:35,394::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=38, domainsMap=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:13:35,395::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`5bf74bd3-dade-404d-bcae-4a5dfe36ef9f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-28::DEBUG::2014-04-11 22:13:35,396::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-28::DEBUG::2014-04-11 22:13:35,396::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-28::DEBUG::2014-04-11 22:13:35,396::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`5bf74bd3-dade-404d-bcae-4a5dfe36ef9f`::Granted request
>Thread-28::DEBUG::2014-04-11 22:13:35,399::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`4f06bab6-01cb-4fbf-8307-932a9ee6318c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-28::DEBUG::2014-04-11 22:13:35,399::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-28::DEBUG::2014-04-11 22:13:35,400::resourceManager::577::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is currently locked, Entering queue (1 in queue)
>Thread-14::DEBUG::2014-04-11 22:13:47,513::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>Thread-14::INFO::2014-04-11 22:13:47,525::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-14::DEBUG::2014-04-11 22:13:47,580::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>Thread-14::INFO::2014-04-11 22:13:47,581::sp::666::Storage.StoragePool::(createMaster) setting master domain for spUUID a106ab81-9d5f-49c1-aeaf-832a137b708c on sdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:13:47,581::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,582::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>Thread-14::DEBUG::2014-04-11 22:13:47,582::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=a033c2ac-0d01-490c-9552-99ca53d6a64a:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2e7a13952d8abc1dc29bd0da89fde82273217f47']
>Thread-14::DEBUG::2014-04-11 22:13:47,611::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=a033c2ac-0d01-490c-9552-99ca53d6a64a:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2e7a13952d8abc1dc29bd0da89fde82273217f47']
>Thread-14::DEBUG::2014-04-11 22:13:47,632::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,632::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,632::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,633::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,633::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,633::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38
>Thread-14::DEBUG::2014-04-11 22:13:47,634::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>Thread-14::DEBUG::2014-04-11 22:13:47,634::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:13:47,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=a033c2ac-0d01-490c-9552-99ca53d6a64a:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2e7a13952d8abc1dc29bd0da89fde82273217f47']
>Thread-14::INFO::2014-04-11 22:13:47,650::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-65::DEBUG::2014-04-11 22:13:47,651::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:13:47,651::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `a033c2ac-0d01-490c-9552-99ca53d6a64a`
>Thread-14::DEBUG::2014-04-11 22:13:47,652::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::INFO::2014-04-11 22:13:47,654::sp::716::Storage.StoragePool::(reconstructMaster) Set storage pool domains: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'Active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'Active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'Active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'Active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'Active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'Active', '21619c8e-99ea-4813-be02-d708971e5393': 'Active'}
>Thread-14::DEBUG::2014-04-11 22:13:47,655::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,656::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>Thread-14::DEBUG::2014-04-11 22:13:47,657::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-14::DEBUG::2014-04-11 22:13:47,673::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>Thread-14::DEBUG::2014-04-11 22:13:47,674::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,674::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,675::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,675::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,675::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,676::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,676::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:13:47,676::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:13:47,677::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 22:13:47,677::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:13:47,677::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 22:13:47,678::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:13:47,728::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:13:47,729::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:13:47,851::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:13:47,852::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,853::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,853::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,854::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,854::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,855::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,855::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::ERROR::2014-04-11 22:13:47,855::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::ERROR::2014-04-11 22:13:47,856::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:13:47,856::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:47,862::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:13:48,198::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:13:48,201::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-14::DEBUG::2014-04-11 22:13:48,202::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:13:48,273::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:13:48,276::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-14::DEBUG::2014-04-11 22:13:48,293::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-14::DEBUG::2014-04-11 22:13:48,296::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-14::INFO::2014-04-11 22:13:48,296::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered
>Thread-14::INFO::2014-04-11 22:13:48,297::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered
>Thread-14::DEBUG::2014-04-11 22:13:48,299::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38
>Thread-14::DEBUG::2014-04-11 22:13:48,301::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>Thread-14::DEBUG::2014-04-11 22:13:48,304::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:13:48,330::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-65::DEBUG::2014-04-11 22:13:48,331::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-14::INFO::2014-04-11 22:13:48,346::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-72::DEBUG::2014-04-11 22:13:48,350::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-14::DEBUG::2014-04-11 22:13:48,350::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `423b56b3-3148-468c-8c28-4c96fddbc1c8`
>Thread-72::ERROR::2014-04-11 22:13:48,355::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-14::INFO::2014-04-11 22:13:48,356::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-72::ERROR::2014-04-11 22:13:48,359::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-73::DEBUG::2014-04-11 22:13:48,366::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-14::DEBUG::2014-04-11 22:13:48,365::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87`
>Thread-72::DEBUG::2014-04-11 22:13:48,369::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::INFO::2014-04-11 22:13:48,373::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-74::DEBUG::2014-04-11 22:13:48,388::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-14::DEBUG::2014-04-11 22:13:48,387::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `b7663d70-e658-41fa-b9f0-8da83c9eddce`
>Thread-74::ERROR::2014-04-11 22:13:48,403::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-14::INFO::2014-04-11 22:13:48,403::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-74::ERROR::2014-04-11 22:13:48,442::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-14::DEBUG::2014-04-11 22:13:48,465::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `3406665e-4adc-4fd4-aa1e-037547b29adb`
>Thread-81::DEBUG::2014-04-11 22:13:48,466::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-74::DEBUG::2014-04-11 22:13:48,474::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-65::DEBUG::2014-04-11 22:13:48,473::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000509115 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14::INFO::2014-04-11 22:13:48,476::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-81::ERROR::2014-04-11 22:13:48,484::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-72::DEBUG::2014-04-11 22:13:48,491::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None)
>Thread-81::ERROR::2014-04-11 22:13:48,493::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-14::DEBUG::2014-04-11 22:13:48,500::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `f3b51811-4a7f-43af-8633-322b3db23c48`
>Thread-82::DEBUG::2014-04-11 22:13:48,513::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-14::INFO::2014-04-11 22:13:48,527::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 21619c8e-99ea-4813-be02-d708971e5393
>Thread-81::DEBUG::2014-04-11 22:13:48,531::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-82::ERROR::2014-04-11 22:13:48,545::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-82::ERROR::2014-04-11 22:13:48,555::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-14::DEBUG::2014-04-11 22:13:48,556::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `21619c8e-99ea-4813-be02-d708971e5393`
>Thread-83::DEBUG::2014-04-11 22:13:48,556::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 21619c8e-99ea-4813-be02-d708971e5393
>Thread-74::DEBUG::2014-04-11 22:13:48,558::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:13:48,567::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-82::DEBUG::2014-04-11 22:13:48,589::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-65::DEBUG::2014-04-11 22:13:48,617::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain a033c2ac-0d01-490c-9552-99ca53d6a64a changed its status to Valid
>Thread-88::DEBUG::2014-04-11 22:13:48,661::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-81::DEBUG::2014-04-11 22:13:48,686::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-88::DEBUG::2014-04-11 22:13:48,700::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-88::DEBUG::2014-04-11 22:13:48,745::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-82::DEBUG::2014-04-11 22:13:48,754::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-91::INFO::2014-04-11 22:13:48,775::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-14::DEBUG::2014-04-11 22:13:48,804::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-72::DEBUG::2014-04-11 22:13:48,815::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5
>Thread-14::DEBUG::2014-04-11 22:13:48,852::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-72::WARNING::2014-04-11 22:13:48,866::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8']
>Thread-74::DEBUG::2014-04-11 22:13:48,905::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5
>Thread-73::DEBUG::2014-04-11 22:13:48,941::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-74::WARNING::2014-04-11 22:13:48,955::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce']
>Thread-14::DEBUG::2014-04-11 22:13:49,053::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8 e2ffd939-380e-4fbb-8a23-5a0e69c84a87 b7663d70-e658-41fa-b9f0-8da83c9eddce 3406665e-4adc-4fd4-aa1e-037547b29adb f3b51811-4a7f-43af-8633-322b3db23c48 21619c8e-99ea-4813-be02-d708971e5393' (cwd None)
>Thread-81::DEBUG::2014-04-11 22:13:49,102::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-82::DEBUG::2014-04-11 22:13:49,179::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-82::WARNING::2014-04-11 22:13:49,198::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-81::WARNING::2014-04-11 22:13:49,198::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-83::DEBUG::2014-04-11 22:13:49,228::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393
>Thread-73::DEBUG::2014-04-11 22:13:49,407::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-14::DEBUG::2014-04-11 22:13:49,489::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n  Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found\n  Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87\n  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n  Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found\n  Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:13:49,492::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8', '  Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found', '  Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87', '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce', '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb', '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48', '  Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found', '  Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393']
>Thread-14::DEBUG::2014-04-11 22:13:49,493::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-73::DEBUG::2014-04-11 22:13:49,590::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_Storage', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=e2ffd939-380e-4fbb-8a23-5a0e69c84a87', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=aa653ed432ff22d61d7b0ce5019a22c0a8bb40dd']
>Thread-73::DEBUG::2014-04-11 22:13:49,621::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-73::DEBUG::2014-04-11 22:13:49,626::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87_imageNS'
>Thread-73::DEBUG::2014-04-11 22:13:49,633::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87_volumeNS'
>Thread-73::DEBUG::2014-04-11 22:13:49,691::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-73::DEBUG::2014-04-11 22:13:49,710::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000299189 s, 1.2 MB/s\n'; <rc> = 0
>Thread-83::DEBUG::2014-04-11 22:13:49,717::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-73::DEBUG::2014-04-11 22:13:49,725::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 changed its status to Valid
>Thread-122::DEBUG::2014-04-11 22:13:49,728::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-122::DEBUG::2014-04-11 22:13:49,729::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-122::DEBUG::2014-04-11 22:13:49,730::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-123::INFO::2014-04-11 22:13:49,732::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-83::DEBUG::2014-04-11 22:13:49,755::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2']
>Thread-83::DEBUG::2014-04-11 22:13:49,760::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-83::DEBUG::2014-04-11 22:13:49,761::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '21619c8e-99ea-4813-be02-d708971e5393_imageNS'
>Thread-83::DEBUG::2014-04-11 22:13:49,762::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '21619c8e-99ea-4813-be02-d708971e5393_volumeNS'
>Thread-83::DEBUG::2014-04-11 22:13:49,816::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-83::DEBUG::2014-04-11 22:13:49,835::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000273719 s, 2.8 MB/s\n'; <rc> = 0
>Thread-83::DEBUG::2014-04-11 22:13:49,841::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 21619c8e-99ea-4813-be02-d708971e5393 changed its status to Valid
>Thread-124::DEBUG::2014-04-11 22:13:49,843::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-124::DEBUG::2014-04-11 22:13:49,843::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-124::DEBUG::2014-04-11 22:13:49,845::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-125::INFO::2014-04-11 22:13:49,846::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-65::DEBUG::2014-04-11 22:13:58,697::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:13:58,720::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000464114 s, 1.6 MB/s\n'; <rc> = 0
>Thread-73::DEBUG::2014-04-11 22:13:59,805::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-73::DEBUG::2014-04-11 22:13:59,823::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000307352 s, 1.2 MB/s\n'; <rc> = 0
>Thread-83::DEBUG::2014-04-11 22:13:59,872::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-83::DEBUG::2014-04-11 22:13:59,892::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00024012 s, 3.1 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:14:08,748::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:14:08,766::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000459543 s, 1.6 MB/s\n'; <rc> = 0
>Thread-73::DEBUG::2014-04-11 22:14:09,880::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-73::DEBUG::2014-04-11 22:14:09,962::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000329968 s, 1.1 MB/s\n'; <rc> = 0
>Thread-83::DEBUG::2014-04-11 22:14:10,005::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-83::DEBUG::2014-04-11 22:14:10,023::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000232803 s, 3.2 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:14:18,794::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:14:18,815::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000355251 s, 2.1 MB/s\n'; <rc> = 0
>Thread-73::DEBUG::2014-04-11 22:14:20,001::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-73::DEBUG::2014-04-11 22:14:20,020::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000234093 s, 1.6 MB/s\n'; <rc> = 0
>Thread-83::DEBUG::2014-04-11 22:14:20,132::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-83::DEBUG::2014-04-11 22:14:20,154::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00024019 s, 3.1 MB/s\n'; <rc> = 0
>Thread-60::WARNING::2014-04-11 22:14:25,948::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-61::WARNING::2014-04-11 22:14:25,952::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-60::DEBUG::2014-04-11 22:14:25,962::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-61::DEBUG::2014-04-11 22:14:25,962::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-65::DEBUG::2014-04-11 22:14:28,844::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:14:28,864::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000589284 s, 1.3 MB/s\n'; <rc> = 0
>Thread-73::DEBUG::2014-04-11 22:14:30,060::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-73::DEBUG::2014-04-11 22:14:30,082::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000281212 s, 1.3 MB/s\n'; <rc> = 0
>Thread-83::DEBUG::2014-04-11 22:14:30,284::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-83::DEBUG::2014-04-11 22:14:30,306::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000212503 s, 3.6 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:14:38,894::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:14:38,914::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000451433 s, 1.7 MB/s\n'; <rc> = 0
>Thread-67::WARNING::2014-04-11 22:14:48,252::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-68::WARNING::2014-04-11 22:14:48,266::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-67::DEBUG::2014-04-11 22:14:48,298::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-68::DEBUG::2014-04-11 22:14:48,333::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-76::WARNING::2014-04-11 22:14:48,466::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-77::WARNING::2014-04-11 22:14:48,475::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-76::DEBUG::2014-04-11 22:14:48,526::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-77::DEBUG::2014-04-11 22:14:48,554::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-85::WARNING::2014-04-11 22:14:48,703::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-86::WARNING::2014-04-11 22:14:48,711::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-85::DEBUG::2014-04-11 22:14:48,757::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-86::DEBUG::2014-04-11 22:14:48,782::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-93::WARNING::2014-04-11 22:14:49,047::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-94::WARNING::2014-04-11 22:14:49,060::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-93::DEBUG::2014-04-11 22:14:49,089::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-94::DEBUG::2014-04-11 22:14:49,108::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-72::ERROR::2014-04-11 22:14:49,136::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-72::ERROR::2014-04-11 22:14:49,160::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-72::DEBUG::2014-04-11 22:14:49,190::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 changed its status to Invalid
>Thread-99::WARNING::2014-04-11 22:14:49,185::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-126::DEBUG::2014-04-11 22:14:49,212::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-100::WARNING::2014-04-11 22:14:49,195::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-99::DEBUG::2014-04-11 22:14:49,221::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-126::DEBUG::2014-04-11 22:14:49,230::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-100::DEBUG::2014-04-11 22:14:49,235::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-126::DEBUG::2014-04-11 22:14:49,255::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-74::ERROR::2014-04-11 22:14:49,259::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-74::ERROR::2014-04-11 22:14:49,278::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-74::DEBUG::2014-04-11 22:14:49,301::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain b7663d70-e658-41fa-b9f0-8da83c9eddce changed its status to Invalid
>Thread-128::DEBUG::2014-04-11 22:14:49,317::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-128::DEBUG::2014-04-11 22:14:49,331::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-128::DEBUG::2014-04-11 22:14:49,352::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-106::WARNING::2014-04-11 22:14:49,386::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-106::DEBUG::2014-04-11 22:14:49,409::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-107::WARNING::2014-04-11 22:14:49,413::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-107::DEBUG::2014-04-11 22:14:49,436::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-109::WARNING::2014-04-11 22:14:49,438::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-108::WARNING::2014-04-11 22:14:49,443::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-109::DEBUG::2014-04-11 22:14:49,456::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-81::ERROR::2014-04-11 22:14:49,465::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-81::ERROR::2014-04-11 22:14:49,467::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-82::ERROR::2014-04-11 22:14:49,468::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-81::DEBUG::2014-04-11 22:14:49,473::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3406665e-4adc-4fd4-aa1e-037547b29adb changed its status to Invalid
>Thread-82::ERROR::2014-04-11 22:14:49,477::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-130::DEBUG::2014-04-11 22:14:49,479::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-82::DEBUG::2014-04-11 22:14:49,482::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain f3b51811-4a7f-43af-8633-322b3db23c48 changed its status to Invalid
>Thread-130::DEBUG::2014-04-11 22:14:49,485::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-130::DEBUG::2014-04-11 22:14:49,488::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-65::DEBUG::2014-04-11 22:14:49,489::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-131::DEBUG::2014-04-11 22:14:49,490::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-131::DEBUG::2014-04-11 22:14:49,493::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-131::DEBUG::2014-04-11 22:14:49,509::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-65::DEBUG::2014-04-11 22:14:49,542::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000409051 s, 1.8 MB/s\n'; <rc> = 0
>Thread-117::WARNING::2014-04-11 22:14:49,580::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-117::DEBUG::2014-04-11 22:14:49,585::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-118::WARNING::2014-04-11 22:14:49,588::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-14::INFO::2014-04-11 22:14:49,589::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-14::INFO::2014-04-11 22:14:49,591::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/21619c8e-99ea-4813-be02-d708971e5393
>Thread-14::INFO::2014-04-11 22:14:49,591::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::INFO::2014-04-11 22:14:49,592::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/mastersd
>Thread-14::INFO::2014-04-11 22:14:49,593::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:14:49,597::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released
>Thread-14::INFO::2014-04-11 22:14:49,597::logUtils::47::dispatcher::(wrapper) Run and protect: reconstructMaster, Return response: None
>Thread-14::DEBUG::2014-04-11 22:14:49,597::task::1185::TaskManager.Task::(prepare) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::finished: None
>Thread-14::DEBUG::2014-04-11 22:14:49,598::task::595::TaskManager.Task::(_updateState) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:14:49,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>}
>Thread-14::DEBUG::2014-04-11 22:14:49,598::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:14:49,599::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:14:49,599::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:14:49,600::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:14:49,600::resourceManager::654::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' has 1 requests in queue. Handling top request.
>Thread-14::DEBUG::2014-04-11 22:14:49,600::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`4f06bab6-01cb-4fbf-8307-932a9ee6318c`::Granted request
>Thread-14::DEBUG::2014-04-11 22:14:49,601::resourceManager::686::ResourceManager::(releaseResource) Request 'Request for Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c - exclusive: granted' was granted
>Thread-28::INFO::2014-04-11 22:14:49,602::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e9c125f38
>Thread-14::DEBUG::2014-04-11 22:14:49,602::task::990::TaskManager.Task::(_decref) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::ref 0 aborting False
>Thread-28::INFO::2014-04-11 22:14:49,603::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 38)
>Thread-28::DEBUG::2014-04-11 22:14:49,604::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,605::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,605::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,605::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,606::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,606::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,606::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-28::DEBUG::2014-04-11 22:14:49,607::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-28::DEBUG::2014-04-11 22:14:49,607::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-28::DEBUG::2014-04-11 22:14:49,607::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-28::DEBUG::2014-04-11 22:14:49,607::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-28::DEBUG::2014-04-11 22:14:49,608::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:14:49,655::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-28::DEBUG::2014-04-11 22:14:49,656::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:14:49,698::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [5f90a775]
>Thread-14::DEBUG::2014-04-11 22:14:49,699::task::595::TaskManager.Task::(_updateState) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:14:49,699::logUtils::44::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=38, options=None)
>Thread-14::DEBUG::2014-04-11 22:14:49,700::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`79d794de-c4a3-46a0-9419-18712d5debb9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '860' at 'refreshStoragePool'
>Thread-14::DEBUG::2014-04-11 22:14:49,701::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'shared'
>Thread-14::DEBUG::2014-04-11 22:14:49,701::resourceManager::577::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is currently locked, Entering queue (1 in queue)
>Thread-28::DEBUG::2014-04-11 22:14:49,785::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-28::DEBUG::2014-04-11 22:14:49,786::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,787::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,787::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,788::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,788::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,789::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,789::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-28::ERROR::2014-04-11 22:14:49,789::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-28::ERROR::2014-04-11 22:14:49,790::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-28::DEBUG::2014-04-11 22:14:49,790::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:49,793::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:14:50,134::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-28::WARNING::2014-04-11 22:14:50,136::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-28::DEBUG::2014-04-11 22:14:50,137::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:50,198::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-28::DEBUG::2014-04-11 22:14:50,201::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-28::DEBUG::2014-04-11 22:14:50,218::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-28::DEBUG::2014-04-11 22:14:50,221::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-28::INFO::2014-04-11 22:14:50,221::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered
>Thread-28::INFO::2014-04-11 22:14:50,221::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered
>Thread-28::DEBUG::2014-04-11 22:14:50,222::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38
>Thread-28::DEBUG::2014-04-11 22:14:50,222::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>Thread-28::DEBUG::2014-04-11 22:14:50,223::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-28::DEBUG::2014-04-11 22:14:50,238::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-28::DEBUG::2014-04-11 22:14:50,239::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-28::WARNING::2014-04-11 22:14:50,240::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c already exists
>Thread-28::DEBUG::2014-04-11 22:14:50,256::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-28::DEBUG::2014-04-11 22:14:50,257::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-28::DEBUG::2014-04-11 22:14:50,258::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8 e2ffd939-380e-4fbb-8a23-5a0e69c84a87 b7663d70-e658-41fa-b9f0-8da83c9eddce 3406665e-4adc-4fd4-aa1e-037547b29adb f3b51811-4a7f-43af-8633-322b3db23c48 21619c8e-99ea-4813-be02-d708971e5393' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:14:50,557::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n  Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found\n  Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87\n  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n  Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found\n  Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393\n'; <rc> = 5
>Thread-28::WARNING::2014-04-11 22:14:50,559::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8', '  Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found', '  Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87', '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce', '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb', '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48', '  Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found', '  Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393']
>Thread-28::DEBUG::2014-04-11 22:14:50,559::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-72::ERROR::2014-04-11 22:14:59,213::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-72::ERROR::2014-04-11 22:14:59,214::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-72::DEBUG::2014-04-11 22:14:59,215::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-72::DEBUG::2014-04-11 22:14:59,216::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None)
>Thread-74::ERROR::2014-04-11 22:14:59,319::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-74::ERROR::2014-04-11 22:14:59,320::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-74::DEBUG::2014-04-11 22:14:59,320::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-74::DEBUG::2014-04-11 22:14:59,321::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None)
>Thread-81::ERROR::2014-04-11 22:14:59,480::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-81::ERROR::2014-04-11 22:14:59,480::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-81::DEBUG::2014-04-11 22:14:59,481::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-81::DEBUG::2014-04-11 22:14:59,482::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-82::ERROR::2014-04-11 22:14:59,494::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-82::ERROR::2014-04-11 22:14:59,495::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-82::DEBUG::2014-04-11 22:14:59,496::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-82::DEBUG::2014-04-11 22:14:59,498::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-72::DEBUG::2014-04-11 22:14:59,524::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5
>Thread-72::WARNING::2014-04-11 22:14:59,526::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8']
>Thread-74::DEBUG::2014-04-11 22:14:59,647::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5
>Thread-74::WARNING::2014-04-11 22:14:59,660::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce']
>Thread-81::DEBUG::2014-04-11 22:14:59,826::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-82::DEBUG::2014-04-11 22:14:59,834::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-82::WARNING::2014-04-11 22:14:59,845::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-81::WARNING::2014-04-11 22:14:59,846::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-81::DEBUG::2014-04-11 22:14:59,863::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:00,217::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:15:00,305::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000484123 s, 1.6 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:15:10,341::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:15:10,362::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000456852 s, 1.7 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:15:20,392::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:15:20,413::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000591778 s, 1.3 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:15:30,443::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:15:30,469::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000451917 s, 1.7 MB/s\n'; <rc> = 0
>Thread-73::ERROR::2014-04-11 22:15:40,113::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-73::DEBUG::2014-04-11 22:15:40,119::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 changed its status to Invalid
>Thread-170::DEBUG::2014-04-11 22:15:40,130::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-73::DEBUG::2014-04-11 22:15:40,133::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-170::DEBUG::2014-04-11 22:15:40,140::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-170::DEBUG::2014-04-11 22:15:40,151::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-83::ERROR::2014-04-11 22:15:40,332::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 21619c8e-99ea-4813-be02-d708971e5393 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
>    fileSD.FileStorageDomain.selftest(self)
>  File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest
>    self.oop.os.statvfs(self.domaindir)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-83::DEBUG::2014-04-11 22:15:40,334::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 21619c8e-99ea-4813-be02-d708971e5393 changed its status to Invalid
>Thread-172::DEBUG::2014-04-11 22:15:40,336::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-83::DEBUG::2014-04-11 22:15:40,338::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-172::DEBUG::2014-04-11 22:15:40,342::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-172::DEBUG::2014-04-11 22:15:40,345::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-65::DEBUG::2014-04-11 22:15:40,530::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:15:40,558::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000537818 s, 1.4 MB/s\n'; <rc> = 0
>Thread-135::WARNING::2014-04-11 22:15:50,186::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-137::WARNING::2014-04-11 22:15:50,189::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-138::WARNING::2014-04-11 22:15:50,220::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-136::WARNING::2014-04-11 22:15:50,241::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-141::WARNING::2014-04-11 22:15:50,612::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-142::WARNING::2014-04-11 22:15:50,619::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-143::WARNING::2014-04-11 22:15:50,619::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-144::WARNING::2014-04-11 22:15:50,620::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-28::INFO::2014-04-11 22:15:50,635::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-28::INFO::2014-04-11 22:15:50,640::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/mastersd
>Thread-65::DEBUG::2014-04-11 22:15:50,643::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:15:50,644::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-28::DEBUG::2014-04-11 22:15:50,671::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-28::DEBUG::2014-04-11 22:15:50,674::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-28::DEBUG::2014-04-11 22:15:50,680::resourceManager::654::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' has 1 requests in queue. Handling top request.
>Thread-28::DEBUG::2014-04-11 22:15:50,685::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`79d794de-c4a3-46a0-9419-18712d5debb9`::Granted request
>Thread-28::DEBUG::2014-04-11 22:15:50,698::resourceManager::686::ResourceManager::(releaseResource) Request 'Request for Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c - shared: granted' was granted
>Thread-28::DEBUG::2014-04-11 22:15:50,707::resourceManager::694::ResourceManager::(releaseResource) This is a shared lock. Granting all shared requests
>Thread-14::DEBUG::2014-04-11 22:15:50,718::task::827::TaskManager.Task::(resourceAcquired) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (shared)
>Thread-28::DEBUG::2014-04-11 22:15:50,726::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 22:15:50,732::task::990::TaskManager.Task::(_decref) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::ref 1 aborting False
>Thread-65::DEBUG::2014-04-11 22:15:50,739::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000395287 s, 1.9 MB/s\n'; <rc> = 0
>Thread-28::DEBUG::2014-04-11 22:15:50,740::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::INFO::2014-04-11 22:15:50,745::fileSD::316::Storage.StorageDomain::(validate) sdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-28::DEBUG::2014-04-11 22:15:50,753::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-28::DEBUG::2014-04-11 22:15:50,766::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-28::INFO::2014-04-11 22:15:50,769::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
>Thread-28::DEBUG::2014-04-11 22:15:50,770::task::1185::TaskManager.Task::(prepare) Task=`bb2d61f2-9971-4bcc-9374-5e2746c7e69f`::finished: True
>Thread-28::DEBUG::2014-04-11 22:15:50,771::task::595::TaskManager.Task::(_updateState) Task=`bb2d61f2-9971-4bcc-9374-5e2746c7e69f`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:15:50,771::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:15:50,772::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:15:50,773::task::990::TaskManager.Task::(_decref) Task=`bb2d61f2-9971-4bcc-9374-5e2746c7e69f`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:15:50,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-14::DEBUG::2014-04-11 22:15:50,815::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:15:50,832::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:15:50,843::task::595::TaskManager.Task::(_updateState) Task=`daa250f8-7b72-499a-aecb-64852eb2cc9b`::moving from state init -> state preparing
>Thread-14::DEBUG::2014-04-11 22:15:50,844::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-28::INFO::2014-04-11 22:15:50,849::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::DEBUG::2014-04-11 22:15:50,855::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:15:50,862::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-28::INFO::2014-04-11 22:15:50,865::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.7', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.7', 'valid': False}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.6', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.4', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000537818', 'lastCheck': '10.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.5', 'valid': False}}
>Thread-28::DEBUG::2014-04-11 22:15:50,870::task::1185::TaskManager.Task::(prepare) Task=`daa250f8-7b72-499a-aecb-64852eb2cc9b`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.7', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.7', 'valid': False}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.6', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.4', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000537818', 'lastCheck': '10.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.5', 'valid': False}}
>Thread-14::DEBUG::2014-04-11 22:15:50,870::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:15:50,875::task::595::TaskManager.Task::(_updateState) Task=`daa250f8-7b72-499a-aecb-64852eb2cc9b`::moving from state preparing -> state finished
>Thread-65::DEBUG::2014-04-11 22:15:50,879::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-28::DEBUG::2014-04-11 22:15:50,882::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-65::DEBUG::2014-04-11 22:15:50,885::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-28::DEBUG::2014-04-11 22:15:50,889::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-65::DEBUG::2014-04-11 22:15:50,891::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-65::DEBUG::2014-04-11 22:15:50,897::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-28::DEBUG::2014-04-11 22:15:50,897::task::990::TaskManager.Task::(_decref) Task=`daa250f8-7b72-499a-aecb-64852eb2cc9b`::ref 0 aborting False
>Thread-65::DEBUG::2014-04-11 22:15:50,903::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-65::DEBUG::2014-04-11 22:15:50,916::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:15:50,968::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-65::DEBUG::2014-04-11 22:15:50,973::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:15:51,003::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:15:51,011::task::595::TaskManager.Task::(_updateState) Task=`c033a710-6744-4135-886d-243cb5599a54`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:15:51,012::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-65::DEBUG::2014-04-11 22:15:51,155::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:15:51,156::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:51,157::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:51,158::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:51,160::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:51,161::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:51,164::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:51,164::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-65::ERROR::2014-04-11 22:15:51,165::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-65::ERROR::2014-04-11 22:15:51,165::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-65::DEBUG::2014-04-11 22:15:51,166::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:51,174::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:15:51,519::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>Thread-65::WARNING::2014-04-11 22:15:51,525::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>Thread-65::DEBUG::2014-04-11 22:15:51,527::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-65::DEBUG::2014-04-11 22:15:51,706::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-65::DEBUG::2014-04-11 22:15:51,711::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-65::DEBUG::2014-04-11 22:15:51,734::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-65::DEBUG::2014-04-11 22:15:51,741::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-65::INFO::2014-04-11 22:15:51,743::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered
>Thread-65::INFO::2014-04-11 22:15:51,744::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered
>Thread-14::DEBUG::2014-04-11 22:15:51,746::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38
>Thread-14::DEBUG::2014-04-11 22:15:51,753::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>Thread-14::DEBUG::2014-04-11 22:15:51,758::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-28::INFO::2014-04-11 22:15:51,765::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-28::DEBUG::2014-04-11 22:15:51,770::task::1185::TaskManager.Task::(prepare) Task=`c033a710-6744-4135-886d-243cb5599a54`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-28::DEBUG::2014-04-11 22:15:51,771::task::595::TaskManager.Task::(_updateState) Task=`c033a710-6744-4135-886d-243cb5599a54`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:15:51,771::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:15:51,772::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:15:51,774::task::990::TaskManager.Task::(_decref) Task=`c033a710-6744-4135-886d-243cb5599a54`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:15:51,793::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-14::DEBUG::2014-04-11 22:15:51,799::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::WARNING::2014-04-11 22:15:51,800::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c already exists
>Thread-14::DEBUG::2014-04-11 22:15:51,822::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-14::DEBUG::2014-04-11 22:15:51,836::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:15:51,911::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8 e2ffd939-380e-4fbb-8a23-5a0e69c84a87 b7663d70-e658-41fa-b9f0-8da83c9eddce 3406665e-4adc-4fd4-aa1e-037547b29adb f3b51811-4a7f-43af-8633-322b3db23c48 21619c8e-99ea-4813-be02-d708971e5393' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:15:51,946::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:15:51,966::task::595::TaskManager.Task::(_updateState) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:15:51,992::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
>Thread-28::DEBUG::2014-04-11 22:15:52,013::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d33fe88e-3805-4d7d-835a-1b29b806c56a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart'
>Thread-28::DEBUG::2014-04-11 22:15:52,021::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-28::DEBUG::2014-04-11 22:15:52,030::resourceManager::577::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is currently locked, Entering queue (1 in queue)
>Thread-14::DEBUG::2014-04-11 22:15:52,225::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n  Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found\n  Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87\n  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n  Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found\n  Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393\n'; <rc> = 5
>Thread-14::WARNING::2014-04-11 22:15:52,233::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8', '  Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found', '  Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87', '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce', '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb', '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48', '  Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found', '  Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393']
>Thread-14::DEBUG::2014-04-11 22:15:52,234::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-147::WARNING::2014-04-11 22:15:59,611::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-148::WARNING::2014-04-11 22:15:59,646::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-150::WARNING::2014-04-11 22:15:59,650::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-149::WARNING::2014-04-11 22:15:59,651::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-150::DEBUG::2014-04-11 22:15:59,744::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-149::DEBUG::2014-04-11 22:15:59,767::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-153::WARNING::2014-04-11 22:15:59,779::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-72::ERROR::2014-04-11 22:15:59,814::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-154::WARNING::2014-04-11 22:15:59,812::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-153::DEBUG::2014-04-11 22:15:59,836::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-72::ERROR::2014-04-11 22:15:59,848::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-155::WARNING::2014-04-11 22:15:59,820::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-154::DEBUG::2014-04-11 22:15:59,862::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-156::WARNING::2014-04-11 22:15:59,822::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-155::DEBUG::2014-04-11 22:15:59,900::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-156::DEBUG::2014-04-11 22:15:59,912::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-74::ERROR::2014-04-11 22:15:59,941::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-74::ERROR::2014-04-11 22:15:59,967::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-159::WARNING::2014-04-11 22:15:59,982::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-159::DEBUG::2014-04-11 22:16:00,018::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-161::DEBUG::2014-04-11 22:16:00,025::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-160::WARNING::2014-04-11 22:16:00,015::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-160::DEBUG::2014-04-11 22:16:00,049::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-161::WARNING::2014-04-11 22:16:00,041::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-161::DEBUG::2014-04-11 22:16:00,067::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-162::WARNING::2014-04-11 22:16:00,081::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-162::DEBUG::2014-04-11 22:16:00,101::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-82::ERROR::2014-04-11 22:16:00,115::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-82::ERROR::2014-04-11 22:16:00,126::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-165::WARNING::2014-04-11 22:16:00,145::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-167::WARNING::2014-04-11 22:16:00,147::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-165::DEBUG::2014-04-11 22:16:00,160::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-167::DEBUG::2014-04-11 22:16:00,165::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-166::WARNING::2014-04-11 22:16:00,162::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-168::WARNING::2014-04-11 22:16:00,169::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-166::DEBUG::2014-04-11 22:16:00,174::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-168::DEBUG::2014-04-11 22:16:00,175::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-81::ERROR::2014-04-11 22:16:00,179::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-81::ERROR::2014-04-11 22:16:00,179::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-65::DEBUG::2014-04-11 22:16:01,788::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:16:01,810::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000445846 s, 1.7 MB/s\n'; <rc> = 0
>Thread-72::ERROR::2014-04-11 22:16:09,890::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-72::ERROR::2014-04-11 22:16:09,891::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-72::DEBUG::2014-04-11 22:16:09,891::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-72::DEBUG::2014-04-11 22:16:09,893::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None)
>Thread-74::ERROR::2014-04-11 22:16:10,001::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-74::ERROR::2014-04-11 22:16:10,001::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-74::DEBUG::2014-04-11 22:16:10,002::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-74::DEBUG::2014-04-11 22:16:10,003::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None)
>Thread-82::ERROR::2014-04-11 22:16:10,142::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-82::ERROR::2014-04-11 22:16:10,142::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-82::DEBUG::2014-04-11 22:16:10,143::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-82::DEBUG::2014-04-11 22:16:10,144::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-81::ERROR::2014-04-11 22:16:10,180::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-81::ERROR::2014-04-11 22:16:10,181::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-81::DEBUG::2014-04-11 22:16:10,182::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-81::DEBUG::2014-04-11 22:16:10,183::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-72::DEBUG::2014-04-11 22:16:10,234::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5
>Thread-72::WARNING::2014-04-11 22:16:10,238::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8']
>Thread-74::DEBUG::2014-04-11 22:16:10,365::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5
>Thread-74::WARNING::2014-04-11 22:16:10,367::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce']
>Thread-82::DEBUG::2014-04-11 22:16:10,504::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-81::DEBUG::2014-04-11 22:16:10,505::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-81::WARNING::2014-04-11 22:16:10,511::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-82::WARNING::2014-04-11 22:16:10,511::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-82::DEBUG::2014-04-11 22:16:10,516::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-65::DEBUG::2014-04-11 22:16:11,840::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:16:11,864::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000356065 s, 2.1 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:16:21,895::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:16:21,918::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.00042857 s, 1.8 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:16:31,949::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:16:31,978::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000454263 s, 1.7 MB/s\n'; <rc> = 0
>Thread-65::DEBUG::2014-04-11 22:16:42,012::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:16:42,042::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000421787 s, 1.8 MB/s\n'; <rc> = 0
>Thread-176::WARNING::2014-04-11 22:16:50,241::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-175::DEBUG::2014-04-11 22:16:50,291::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-177::DEBUG::2014-04-11 22:16:50,292::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-178::WARNING::2014-04-11 22:16:50,283::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-176::DEBUG::2014-04-11 22:16:50,336::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-178::DEBUG::2014-04-11 22:16:50,362::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-175::WARNING::2014-04-11 22:16:50,342::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-177::WARNING::2014-04-11 22:16:50,351::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-175::DEBUG::2014-04-11 22:16:50,417::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-177::DEBUG::2014-04-11 22:16:50,449::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-183::WARNING::2014-04-11 22:16:50,439::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-73::ERROR::2014-04-11 22:16:50,488::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',)
>Thread-182::WARNING::2014-04-11 22:16:50,441::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-183::DEBUG::2014-04-11 22:16:50,498::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-181::WARNING::2014-04-11 22:16:50,463::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-73::ERROR::2014-04-11 22:16:50,520::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>    return getattr(self.getRealDomain(), attrName)
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',)
>Thread-184::WARNING::2014-04-11 22:16:50,468::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-182::DEBUG::2014-04-11 22:16:50,531::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-181::DEBUG::2014-04-11 22:16:50,547::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-184::DEBUG::2014-04-11 22:16:50,575::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-83::ERROR::2014-04-11 22:16:50,602::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 21619c8e-99ea-4813-be02-d708971e5393 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',)
>Thread-83::ERROR::2014-04-11 22:16:50,623::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 21619c8e-99ea-4813-be02-d708971e5393 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>    return getattr(self.getRealDomain(), attrName)
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',)
>Thread-187::WARNING::2014-04-11 22:16:51,642::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-187::DEBUG::2014-04-11 22:16:51,674::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-188::WARNING::2014-04-11 22:16:51,674::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-189::WARNING::2014-04-11 22:16:51,681::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-188::DEBUG::2014-04-11 22:16:51,710::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-190::WARNING::2014-04-11 22:16:51,686::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-189::DEBUG::2014-04-11 22:16:51,722::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-190::DEBUG::2014-04-11 22:16:51,738::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-193::WARNING::2014-04-11 22:16:52,346::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-193::DEBUG::2014-04-11 22:16:52,374::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-194::WARNING::2014-04-11 22:16:52,383::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-195::WARNING::2014-04-11 22:16:52,387::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-194::DEBUG::2014-04-11 22:16:52,405::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-196::WARNING::2014-04-11 22:16:52,392::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-195::DEBUG::2014-04-11 22:16:52,416::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-196::DEBUG::2014-04-11 22:16:52,426::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-14::INFO::2014-04-11 22:16:52,443::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::INFO::2014-04-11 22:16:52,450::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/mastersd
>Thread-14::INFO::2014-04-11 22:16:52,454::logUtils::47::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None
>Thread-14::DEBUG::2014-04-11 22:16:52,457::task::1185::TaskManager.Task::(prepare) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::finished: None
>Thread-14::DEBUG::2014-04-11 22:16:52,462::task::595::TaskManager.Task::(_updateState) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:16:52,465::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>}
>Thread-14::DEBUG::2014-04-11 22:16:52,469::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:16:52,473::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:16:52,480::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:16:52,483::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:16:52,485::resourceManager::654::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' has 1 requests in queue. Handling top request.
>Thread-14::DEBUG::2014-04-11 22:16:52,489::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d33fe88e-3805-4d7d-835a-1b29b806c56a`::Granted request
>Thread-14::DEBUG::2014-04-11 22:16:52,493::resourceManager::686::ResourceManager::(releaseResource) Request 'Request for Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c - exclusive: granted' was granted
>Thread-28::DEBUG::2014-04-11 22:16:52,502::task::827::TaskManager.Task::(resourceAcquired) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive)
>Thread-14::DEBUG::2014-04-11 22:16:52,502::task::990::TaskManager.Task::(_decref) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:16:52,508::task::990::TaskManager.Task::(_decref) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::ref 1 aborting False
>Thread-28::INFO::2014-04-11 22:16:52,515::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
>Thread-28::DEBUG::2014-04-11 22:16:52,520::task::1180::TaskManager.Task::(prepare) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::Prepare: 1 jobs exist, move to acquiring
>Thread-28::DEBUG::2014-04-11 22:16:52,522::task::595::TaskManager.Task::(_updateState) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::moving from state preparing -> state acquiring
>Thread-28::DEBUG::2014-04-11 22:16:52,527::task::595::TaskManager.Task::(_updateState) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::moving from state acquiring -> state queued
>Thread-28::DEBUG::2014-04-11 22:16:52,531::taskManager::50::TaskManager::(_queueTask) queuing task: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:16:52,537::taskManager::56::TaskManager::(_queueTask) task queued: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-6::DEBUG::2014-04-11 22:16:52,538::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
>Thread-28::DEBUG::2014-04-11 22:16:52,542::task::1182::TaskManager.Task::(prepare) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::returning
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,543::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 running: <bound method Task.commit of <storage.task.Task instance at 0x7f4e840830e0>> with: None
>Thread-28::DEBUG::2014-04-11 22:16:52,548::task::990::TaskManager.Task::(_decref) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::ref 0 aborting False
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,549::task::1194::TaskManager.Task::(commit) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::committing task: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,556::task::595::TaskManager.Task::(_updateState) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::moving from state queued -> state running
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,562::task::905::TaskManager.Task::(_runJobs) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f4e6802c6d0>> (args: (-1, '-1', 250, 3) kwargs: {})
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,566::task::331::TaskManager.Task::(run) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f4e6802c6d0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,570::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,573::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-65::DEBUG::2014-04-11 22:16:52,586::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,611::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,617::misc::772::SamplingMethod::(__call__) Returning last result
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:52,639::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,646::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>Thread-65::DEBUG::2014-04-11 22:16:52,655::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000528242 s, 1.4 MB/s\n'; <rc> = 0
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:52,659::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-28::DEBUG::2014-04-11 22:16:52,675::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [2860bcb8]
>Thread-28::DEBUG::2014-04-11 22:16:52,682::task::595::TaskManager.Task::(_updateState) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:16:52,693::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, remove=False, options=None)
>Thread-28::ERROR::2014-04-11 22:16:52,700::task::866::TaskManager.Task::(_setError) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 1119, in disconnectStoragePool
>    self.validateNotSPM(spUUID)
>  File "/usr/share/vdsm/storage/hsm.py", line 320, in validateNotSPM
>    raise se.IsSpm(spUUID)
>IsSpm: Operation not allowed while SPM is active: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)
>Thread-28::DEBUG::2014-04-11 22:16:52,713::task::885::TaskManager.Task::(_run) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::Task._run: 6531d861-ec83-4492-8e4b-0673916f1136 ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, False) {} failed - stopping task
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,717::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>Thread-28::DEBUG::2014-04-11 22:16:52,721::task::1211::TaskManager.Task::(stop) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::stopping in state preparing (force False)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,726::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
>Thread-28::DEBUG::2014-04-11 22:16:52,732::task::990::TaskManager.Task::(_decref) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::ref 1 aborting True
>Thread-28::INFO::2014-04-11 22:16:52,740::task::1168::TaskManager.Task::(prepare) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::aborting: Task is aborted: 'Operation not allowed while SPM is active' - code 656
>Thread-28::DEBUG::2014-04-11 22:16:52,746::task::1173::TaskManager.Task::(prepare) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::Prepare: aborted: Operation not allowed while SPM is active
>Thread-28::DEBUG::2014-04-11 22:16:52,751::task::990::TaskManager.Task::(_decref) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::ref 0 aborting True
>Thread-28::DEBUG::2014-04-11 22:16:52,755::task::925::TaskManager.Task::(_doAbort) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::Task._doAbort: force False
>Thread-28::DEBUG::2014-04-11 22:16:52,760::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:16:52,766::task::595::TaskManager.Task::(_updateState) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::moving from state preparing -> state aborting
>Thread-28::DEBUG::2014-04-11 22:16:52,774::task::550::TaskManager.Task::(__state_aborting) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::_aborting: recover policy none
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,774::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169']
>Thread-28::DEBUG::2014-04-11 22:16:52,781::task::595::TaskManager.Task::(_updateState) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::moving from state aborting -> state failed
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,786::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>Thread-28::DEBUG::2014-04-11 22:16:52,787::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:16:52,796::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::ERROR::2014-04-11 22:16:52,802::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Operation not allowed while SPM is active: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 656}}
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,804::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,816::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,864::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,873::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`7c9c6e9a-712e-4218-a35a-03bc61a13e3a`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,878::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,884::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,889::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`7c9c6e9a-712e-4218-a35a-03bc61a13e3a`::Granted request
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:52,898::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a`
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,907::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`1f88c4c9-6777-4dd4-9b40-b052c87a98e0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,910::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive'
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,917::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,922::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`1f88c4c9-6777-4dd4-9b40-b052c87a98e0`::Granted request
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::ERROR::2014-04-11 22:16:52,933::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::ERROR::2014-04-11 22:16:52,937::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,942::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,962::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,256::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::WARNING::2014-04-11 22:16:53,258::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', '  Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a']
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,258::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,394::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,396::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,413::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,415::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:53,416::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:53,416::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,417::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a'
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,417::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,417::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it.
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,418::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records.
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,418::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,434::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,435::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,435::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,473::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,605::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,662::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-28::DEBUG::2014-04-11 22:16:53,690::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,710::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,734::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms
>Thread-28::DEBUG::2014-04-11 22:16:53,735::task::595::TaskManager.Task::(_updateState) Task=`9c288f0f-dd95-46a1-a93a-de30dc2a7166`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:16:53,737::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,737::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks
>Thread-28::DEBUG::2014-04-11 22:16:53,737::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:16:53,739::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:16:53,740::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:53,741::task::1185::TaskManager.Task::(prepare) Task=`9c288f0f-dd95-46a1-a93a-de30dc2a7166`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:53,741::task::595::TaskManager.Task::(_updateState) Task=`9c288f0f-dd95-46a1-a93a-de30dc2a7166`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:16:53,742::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:16:53,742::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:16:53,743::task::990::TaskManager.Task::(_decref) Task=`9c288f0f-dd95-46a1-a93a-de30dc2a7166`::ref 0 aborting False
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,756::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>Thread-28::DEBUG::2014-04-11 22:16:54,749::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:16:54,750::task::595::TaskManager.Task::(_updateState) Task=`32fbd794-660c-49cb-a034-6e7720a8b515`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:16:54,751::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:16:54,751::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:16:54,752::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:16:54,752::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:54,752::task::1185::TaskManager.Task::(prepare) Task=`32fbd794-660c-49cb-a034-6e7720a8b515`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:54,752::task::595::TaskManager.Task::(_updateState) Task=`32fbd794-660c-49cb-a034-6e7720a8b515`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:16:54,753::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:16:54,753::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:16:54,753::task::990::TaskManager.Task::(_decref) Task=`32fbd794-660c-49cb-a034-6e7720a8b515`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:16:55,758::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:16:55,759::task::595::TaskManager.Task::(_updateState) Task=`0e8c79dc-4842-4eef-b3f3-6e41e91a24ff`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:16:55,759::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:16:55,759::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:16:55,760::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:16:55,760::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:55,761::task::1185::TaskManager.Task::(prepare) Task=`0e8c79dc-4842-4eef-b3f3-6e41e91a24ff`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:55,761::task::595::TaskManager.Task::(_updateState) Task=`0e8c79dc-4842-4eef-b3f3-6e41e91a24ff`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:16:55,761::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:16:55,762::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:16:55,762::task::990::TaskManager.Task::(_decref) Task=`0e8c79dc-4842-4eef-b3f3-6e41e91a24ff`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:16:56,767::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:16:56,767::task::595::TaskManager.Task::(_updateState) Task=`80040a28-cb48-46b3-870a-424dd347b05e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:16:56,768::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:16:56,768::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:16:56,769::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:16:56,769::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:56,769::task::1185::TaskManager.Task::(prepare) Task=`80040a28-cb48-46b3-870a-424dd347b05e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:56,769::task::595::TaskManager.Task::(_updateState) Task=`80040a28-cb48-46b3-870a-424dd347b05e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:16:56,770::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:16:56,771::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:16:56,771::task::990::TaskManager.Task::(_decref) Task=`80040a28-cb48-46b3-870a-424dd347b05e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:16:57,775::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:16:57,776::task::595::TaskManager.Task::(_updateState) Task=`69352223-4797-4ac0-a99a-4b3ac120ca30`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:16:57,776::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:16:57,777::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:16:57,777::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:16:57,778::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:57,778::task::1185::TaskManager.Task::(prepare) Task=`69352223-4797-4ac0-a99a-4b3ac120ca30`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:57,778::task::595::TaskManager.Task::(_updateState) Task=`69352223-4797-4ac0-a99a-4b3ac120ca30`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:16:57,778::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:16:57,779::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:16:57,779::task::990::TaskManager.Task::(_decref) Task=`69352223-4797-4ac0-a99a-4b3ac120ca30`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:16:58,784::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:16:58,787::task::595::TaskManager.Task::(_updateState) Task=`9a2cba63-7bf9-4f6f-a7fe-78f6deb42f21`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:16:58,788::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:16:58,789::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:16:58,791::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:16:58,792::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:58,792::task::1185::TaskManager.Task::(prepare) Task=`9a2cba63-7bf9-4f6f-a7fe-78f6deb42f21`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:58,794::task::595::TaskManager.Task::(_updateState) Task=`9a2cba63-7bf9-4f6f-a7fe-78f6deb42f21`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:16:58,794::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:16:58,794::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:16:58,795::task::990::TaskManager.Task::(_decref) Task=`9a2cba63-7bf9-4f6f-a7fe-78f6deb42f21`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:16:59,800::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:16:59,801::task::595::TaskManager.Task::(_updateState) Task=`c1056ca2-752e-4a03-8d6e-1d30dd5226e9`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:16:59,801::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:16:59,801::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:16:59,802::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:16:59,802::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:59,803::task::1185::TaskManager.Task::(prepare) Task=`c1056ca2-752e-4a03-8d6e-1d30dd5226e9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:16:59,803::task::595::TaskManager.Task::(_updateState) Task=`c1056ca2-752e-4a03-8d6e-1d30dd5226e9`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:16:59,803::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:16:59,804::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:16:59,804::task::990::TaskManager.Task::(_decref) Task=`c1056ca2-752e-4a03-8d6e-1d30dd5226e9`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:00,808::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:00,809::task::595::TaskManager.Task::(_updateState) Task=`f4851d03-4282-4a2f-9dd4-4e6f038159dc`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:00,809::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:00,809::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:00,810::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:00,810::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:00,811::task::1185::TaskManager.Task::(prepare) Task=`f4851d03-4282-4a2f-9dd4-4e6f038159dc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:00,811::task::595::TaskManager.Task::(_updateState) Task=`f4851d03-4282-4a2f-9dd4-4e6f038159dc`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:00,811::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:00,812::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:00,812::task::990::TaskManager.Task::(_decref) Task=`f4851d03-4282-4a2f-9dd4-4e6f038159dc`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:01,817::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:01,817::task::595::TaskManager.Task::(_updateState) Task=`9992bda9-0eef-4500-b1fd-f3b7fe31797c`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:01,818::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:01,818::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:01,819::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:01,819::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:01,820::task::1185::TaskManager.Task::(prepare) Task=`9992bda9-0eef-4500-b1fd-f3b7fe31797c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:01,820::task::595::TaskManager.Task::(_updateState) Task=`9992bda9-0eef-4500-b1fd-f3b7fe31797c`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:01,820::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:01,821::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:01,821::task::990::TaskManager.Task::(_decref) Task=`9992bda9-0eef-4500-b1fd-f3b7fe31797c`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:02,826::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:02,839::task::595::TaskManager.Task::(_updateState) Task=`efdc6dc5-510f-4c13-afc6-494ec03becf1`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:02,840::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:02,840::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:02,841::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:02,841::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:02,841::task::1185::TaskManager.Task::(prepare) Task=`efdc6dc5-510f-4c13-afc6-494ec03becf1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:02,842::task::595::TaskManager.Task::(_updateState) Task=`efdc6dc5-510f-4c13-afc6-494ec03becf1`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:02,842::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:02,843::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:02,843::task::990::TaskManager.Task::(_decref) Task=`efdc6dc5-510f-4c13-afc6-494ec03becf1`::ref 0 aborting False
>Thread-65::DEBUG::2014-04-11 22:17:02,898::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:17:02,926::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000423116 s, 1.8 MB/s\n'; <rc> = 0
>Thread-28::DEBUG::2014-04-11 22:17:03,851::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:03,859::task::595::TaskManager.Task::(_updateState) Task=`22d3c00a-17b3-4ca2-ac11-874e2523f216`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:03,862::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:03,863::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:03,866::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:03,867::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:03,867::task::1185::TaskManager.Task::(prepare) Task=`22d3c00a-17b3-4ca2-ac11-874e2523f216`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:03,867::task::595::TaskManager.Task::(_updateState) Task=`22d3c00a-17b3-4ca2-ac11-874e2523f216`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:03,872::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:03,873::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:03,874::task::990::TaskManager.Task::(_decref) Task=`22d3c00a-17b3-4ca2-ac11-874e2523f216`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:04,886::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:04,893::task::595::TaskManager.Task::(_updateState) Task=`767f8de3-46ac-4f54-be99-bbc96114b067`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:04,894::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:04,897::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:04,897::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:04,901::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:04,903::task::1185::TaskManager.Task::(prepare) Task=`767f8de3-46ac-4f54-be99-bbc96114b067`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:04,904::task::595::TaskManager.Task::(_updateState) Task=`767f8de3-46ac-4f54-be99-bbc96114b067`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:04,909::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:04,910::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:04,911::task::990::TaskManager.Task::(_decref) Task=`767f8de3-46ac-4f54-be99-bbc96114b067`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:05,921::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:05,922::task::595::TaskManager.Task::(_updateState) Task=`725875c3-5b40-496c-9eec-1c0ef4443a2e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:05,923::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:05,923::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:05,924::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:05,924::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:05,925::task::1185::TaskManager.Task::(prepare) Task=`725875c3-5b40-496c-9eec-1c0ef4443a2e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:05,925::task::595::TaskManager.Task::(_updateState) Task=`725875c3-5b40-496c-9eec-1c0ef4443a2e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:05,926::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:05,926::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:05,927::task::990::TaskManager.Task::(_decref) Task=`725875c3-5b40-496c-9eec-1c0ef4443a2e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:06,931::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:06,933::task::595::TaskManager.Task::(_updateState) Task=`471f94b3-32bd-40ad-8691-37ea9c71609e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:06,933::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:06,934::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:06,934::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:06,935::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:06,936::task::1185::TaskManager.Task::(prepare) Task=`471f94b3-32bd-40ad-8691-37ea9c71609e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:06,937::task::595::TaskManager.Task::(_updateState) Task=`471f94b3-32bd-40ad-8691-37ea9c71609e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:06,937::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:06,937::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:06,938::task::990::TaskManager.Task::(_decref) Task=`471f94b3-32bd-40ad-8691-37ea9c71609e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:07,943::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:07,958::task::595::TaskManager.Task::(_updateState) Task=`c3636159-aa3c-4fe9-b9da-55bfc1e93c9f`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:07,959::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:07,960::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:07,960::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:07,961::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:07,962::task::1185::TaskManager.Task::(prepare) Task=`c3636159-aa3c-4fe9-b9da-55bfc1e93c9f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:07,963::task::595::TaskManager.Task::(_updateState) Task=`c3636159-aa3c-4fe9-b9da-55bfc1e93c9f`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:07,963::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:07,964::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:07,965::task::990::TaskManager.Task::(_decref) Task=`c3636159-aa3c-4fe9-b9da-55bfc1e93c9f`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:08,972::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:08,978::task::595::TaskManager.Task::(_updateState) Task=`a658eef8-ad6d-4951-a696-8151371ba65d`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:08,981::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:08,982::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:08,984::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:08,986::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:08,988::task::1185::TaskManager.Task::(prepare) Task=`a658eef8-ad6d-4951-a696-8151371ba65d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:08,990::task::595::TaskManager.Task::(_updateState) Task=`a658eef8-ad6d-4951-a696-8151371ba65d`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:08,991::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:08,993::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:08,995::task::990::TaskManager.Task::(_decref) Task=`a658eef8-ad6d-4951-a696-8151371ba65d`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:10,016::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:10,031::task::595::TaskManager.Task::(_updateState) Task=`9c0bdede-a9e6-4d87-93ca-f5ac92f24e92`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:10,040::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:10,040::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:10,056::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:10,060::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:10,063::task::1185::TaskManager.Task::(prepare) Task=`9c0bdede-a9e6-4d87-93ca-f5ac92f24e92`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:10,070::task::595::TaskManager.Task::(_updateState) Task=`9c0bdede-a9e6-4d87-93ca-f5ac92f24e92`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:10,074::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:10,077::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:10,084::task::990::TaskManager.Task::(_decref) Task=`9c0bdede-a9e6-4d87-93ca-f5ac92f24e92`::ref 0 aborting False
>Thread-199::WARNING::2014-04-11 22:17:10,339::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-200::WARNING::2014-04-11 22:17:10,369::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-199::DEBUG::2014-04-11 22:17:10,413::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-202::WARNING::2014-04-11 22:17:10,374::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-200::DEBUG::2014-04-11 22:17:10,444::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-201::WARNING::2014-04-11 22:17:10,374::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-202::DEBUG::2014-04-11 22:17:10,476::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-205::WARNING::2014-04-11 22:17:10,472::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-201::DEBUG::2014-04-11 22:17:10,506::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-206::WARNING::2014-04-11 22:17:10,495::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-205::DEBUG::2014-04-11 22:17:10,532::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-72::ERROR::2014-04-11 22:17:10,549::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-207::WARNING::2014-04-11 22:17:10,502::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-206::DEBUG::2014-04-11 22:17:10,558::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-208::WARNING::2014-04-11 22:17:10,503::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-72::ERROR::2014-04-11 22:17:10,585::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-228::ERROR::2014-04-11 22:17:10,588::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-207::DEBUG::2014-04-11 22:17:10,598::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-208::DEBUG::2014-04-11 22:17:10,611::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-228::ERROR::2014-04-11 22:17:10,635::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-211::WARNING::2014-04-11 22:17:10,637::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-74::ERROR::2014-04-11 22:17:10,662::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-228::DEBUG::2014-04-11 22:17:10,670::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-211::DEBUG::2014-04-11 22:17:10,683::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-212::WARNING::2014-04-11 22:17:10,668::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-74::ERROR::2014-04-11 22:17:10,695::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-230::ERROR::2014-04-11 22:17:10,696::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-213::WARNING::2014-04-11 22:17:10,679::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-212::DEBUG::2014-04-11 22:17:10,716::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-214::WARNING::2014-04-11 22:17:10,714::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-230::ERROR::2014-04-11 22:17:10,731::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-213::DEBUG::2014-04-11 22:17:10,740::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-214::DEBUG::2014-04-11 22:17:10,751::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-230::DEBUG::2014-04-11 22:17:10,762::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-81::ERROR::2014-04-11 22:17:10,778::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-217::WARNING::2014-04-11 22:17:10,780::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-228::DEBUG::2014-04-11 22:17:10,796::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None)
>Thread-81::ERROR::2014-04-11 22:17:10,802::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-231::ERROR::2014-04-11 22:17:10,803::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-217::DEBUG::2014-04-11 22:17:10,834::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-218::WARNING::2014-04-11 22:17:10,807::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-219::WARNING::2014-04-11 22:17:10,809::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-231::ERROR::2014-04-11 22:17:10,850::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-220::WARNING::2014-04-11 22:17:10,811::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-218::DEBUG::2014-04-11 22:17:10,862::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-219::DEBUG::2014-04-11 22:17:10,865::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-230::DEBUG::2014-04-11 22:17:10,872::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None)
>Thread-231::DEBUG::2014-04-11 22:17:10,872::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-220::DEBUG::2014-04-11 22:17:10,874::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-82::ERROR::2014-04-11 22:17:10,909::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-82::ERROR::2014-04-11 22:17:10,918::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-232::ERROR::2014-04-11 22:17:10,920::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-232::ERROR::2014-04-11 22:17:10,933::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-231::DEBUG::2014-04-11 22:17:10,940::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-232::DEBUG::2014-04-11 22:17:10,946::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-232::DEBUG::2014-04-11 22:17:10,976::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:17:11,112::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:11,113::task::595::TaskManager.Task::(_updateState) Task=`9643246b-0319-44ef-9775-9e99cf35739d`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:11,113::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:11,114::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:11,115::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-228::DEBUG::2014-04-11 22:17:11,116::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5
>Thread-28::INFO::2014-04-11 22:17:11,134::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:11,138::task::1185::TaskManager.Task::(prepare) Task=`9643246b-0319-44ef-9775-9e99cf35739d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:11,140::task::595::TaskManager.Task::(_updateState) Task=`9643246b-0319-44ef-9775-9e99cf35739d`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:11,142::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:11,143::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-228::WARNING::2014-04-11 22:17:11,144::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8']
>Thread-28::DEBUG::2014-04-11 22:17:11,144::task::990::TaskManager.Task::(_decref) Task=`9643246b-0319-44ef-9775-9e99cf35739d`::ref 0 aborting False
>Thread-230::DEBUG::2014-04-11 22:17:11,266::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5
>Thread-231::DEBUG::2014-04-11 22:17:11,275::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-232::DEBUG::2014-04-11 22:17:11,275::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-230::WARNING::2014-04-11 22:17:11,280::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce']
>Thread-232::WARNING::2014-04-11 22:17:11,280::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-231::WARNING::2014-04-11 22:17:11,280::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-231::DEBUG::2014-04-11 22:17:11,287::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:17:12,157::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:12,158::task::595::TaskManager.Task::(_updateState) Task=`e15eb6c4-5e0a-4744-b8da-8c87be236f29`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:12,159::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:12,159::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:12,160::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:12,160::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:12,161::task::1185::TaskManager.Task::(prepare) Task=`e15eb6c4-5e0a-4744-b8da-8c87be236f29`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:12,161::task::595::TaskManager.Task::(_updateState) Task=`e15eb6c4-5e0a-4744-b8da-8c87be236f29`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:12,162::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:12,162::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:12,162::task::990::TaskManager.Task::(_decref) Task=`e15eb6c4-5e0a-4744-b8da-8c87be236f29`::ref 0 aborting False
>Thread-65::DEBUG::2014-04-11 22:17:12,958::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:17:13,009::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000366744 s, 2.1 MB/s\n'; <rc> = 0
>Thread-28::DEBUG::2014-04-11 22:17:13,171::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:13,175::task::595::TaskManager.Task::(_updateState) Task=`2140cbba-f0c6-4d9e-8252-d8e7dbf2034d`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:13,177::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:13,179::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:13,184::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:13,187::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:13,189::task::1185::TaskManager.Task::(prepare) Task=`2140cbba-f0c6-4d9e-8252-d8e7dbf2034d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:13,192::task::595::TaskManager.Task::(_updateState) Task=`2140cbba-f0c6-4d9e-8252-d8e7dbf2034d`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:13,198::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:13,201::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:13,203::task::990::TaskManager.Task::(_decref) Task=`2140cbba-f0c6-4d9e-8252-d8e7dbf2034d`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:14,214::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:14,216::task::595::TaskManager.Task::(_updateState) Task=`d2cde899-4c6e-4f88-b2c7-01f2db97a78a`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:14,217::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:14,217::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:14,217::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:14,218::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:14,218::task::1185::TaskManager.Task::(prepare) Task=`d2cde899-4c6e-4f88-b2c7-01f2db97a78a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:14,218::task::595::TaskManager.Task::(_updateState) Task=`d2cde899-4c6e-4f88-b2c7-01f2db97a78a`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:14,219::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:14,219::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:14,221::task::990::TaskManager.Task::(_decref) Task=`d2cde899-4c6e-4f88-b2c7-01f2db97a78a`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:15,226::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:15,227::task::595::TaskManager.Task::(_updateState) Task=`aedde964-e4b5-4e2a-bce4-3081f1c2d995`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:15,228::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:15,228::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:15,228::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:15,229::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:15,229::task::1185::TaskManager.Task::(prepare) Task=`aedde964-e4b5-4e2a-bce4-3081f1c2d995`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:15,229::task::595::TaskManager.Task::(_updateState) Task=`aedde964-e4b5-4e2a-bce4-3081f1c2d995`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:15,230::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:15,230::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:15,230::task::990::TaskManager.Task::(_decref) Task=`aedde964-e4b5-4e2a-bce4-3081f1c2d995`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:16,234::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:16,235::task::595::TaskManager.Task::(_updateState) Task=`00a4e8a2-bb96-48a8-8d11-45105df110d6`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:16,235::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:16,235::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:16,236::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:16,236::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:16,237::task::1185::TaskManager.Task::(prepare) Task=`00a4e8a2-bb96-48a8-8d11-45105df110d6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:16,237::task::595::TaskManager.Task::(_updateState) Task=`00a4e8a2-bb96-48a8-8d11-45105df110d6`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:16,237::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:16,238::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:16,238::task::990::TaskManager.Task::(_decref) Task=`00a4e8a2-bb96-48a8-8d11-45105df110d6`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:17,242::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:17,243::task::595::TaskManager.Task::(_updateState) Task=`b2eda07b-3d85-4d8a-aa63-f2609c9f0aef`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:17,243::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:17,244::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:17,244::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:17,244::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:17,245::task::1185::TaskManager.Task::(prepare) Task=`b2eda07b-3d85-4d8a-aa63-f2609c9f0aef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:17,245::task::595::TaskManager.Task::(_updateState) Task=`b2eda07b-3d85-4d8a-aa63-f2609c9f0aef`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:17,246::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:17,246::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:17,246::task::990::TaskManager.Task::(_decref) Task=`b2eda07b-3d85-4d8a-aa63-f2609c9f0aef`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:18,250::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:18,251::task::595::TaskManager.Task::(_updateState) Task=`221f94e3-c676-43e2-9977-ffa70f10549c`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:18,252::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:18,252::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:18,252::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:18,253::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:18,253::task::1185::TaskManager.Task::(prepare) Task=`221f94e3-c676-43e2-9977-ffa70f10549c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:18,253::task::595::TaskManager.Task::(_updateState) Task=`221f94e3-c676-43e2-9977-ffa70f10549c`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:18,254::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:18,254::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:18,254::task::990::TaskManager.Task::(_decref) Task=`221f94e3-c676-43e2-9977-ffa70f10549c`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:19,259::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:19,261::task::595::TaskManager.Task::(_updateState) Task=`696c9964-74ac-4de8-8f2e-2e4434a2da59`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:19,261::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:19,262::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:19,262::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:19,262::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:19,263::task::1185::TaskManager.Task::(prepare) Task=`696c9964-74ac-4de8-8f2e-2e4434a2da59`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:19,264::task::595::TaskManager.Task::(_updateState) Task=`696c9964-74ac-4de8-8f2e-2e4434a2da59`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:19,264::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:19,265::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:19,265::task::990::TaskManager.Task::(_decref) Task=`696c9964-74ac-4de8-8f2e-2e4434a2da59`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:20,269::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:20,270::task::595::TaskManager.Task::(_updateState) Task=`9061c9eb-2c40-4581-bb23-6d23f76f0eca`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:20,270::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:20,271::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:20,271::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:20,271::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:20,272::task::1185::TaskManager.Task::(prepare) Task=`9061c9eb-2c40-4581-bb23-6d23f76f0eca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:20,272::task::595::TaskManager.Task::(_updateState) Task=`9061c9eb-2c40-4581-bb23-6d23f76f0eca`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:20,272::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:20,273::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:20,273::task::990::TaskManager.Task::(_decref) Task=`9061c9eb-2c40-4581-bb23-6d23f76f0eca`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:21,278::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:21,280::task::595::TaskManager.Task::(_updateState) Task=`1bcb6b0f-f6ef-45f2-aaee-4434d5eb7763`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:21,281::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:21,281::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:21,281::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:21,282::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:21,282::task::1185::TaskManager.Task::(prepare) Task=`1bcb6b0f-f6ef-45f2-aaee-4434d5eb7763`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:21,282::task::595::TaskManager.Task::(_updateState) Task=`1bcb6b0f-f6ef-45f2-aaee-4434d5eb7763`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:21,283::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:21,283::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:21,284::task::990::TaskManager.Task::(_decref) Task=`1bcb6b0f-f6ef-45f2-aaee-4434d5eb7763`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:22,288::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:22,288::task::595::TaskManager.Task::(_updateState) Task=`3e02b295-9a2d-4101-831c-0d5c63eb4f41`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:22,289::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:22,289::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:22,289::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:22,290::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:22,290::task::1185::TaskManager.Task::(prepare) Task=`3e02b295-9a2d-4101-831c-0d5c63eb4f41`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:22,290::task::595::TaskManager.Task::(_updateState) Task=`3e02b295-9a2d-4101-831c-0d5c63eb4f41`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:22,291::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:22,291::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:22,291::task::990::TaskManager.Task::(_decref) Task=`3e02b295-9a2d-4101-831c-0d5c63eb4f41`::ref 0 aborting False
>Thread-65::DEBUG::2014-04-11 22:17:23,056::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:17:23,079::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000389197 s, 1.9 MB/s\n'; <rc> = 0
>Thread-28::DEBUG::2014-04-11 22:17:23,299::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:23,303::task::595::TaskManager.Task::(_updateState) Task=`de327f1b-eb84-458f-9d22-00e658170711`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:23,306::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:23,307::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:23,309::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:23,311::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:23,314::task::1185::TaskManager.Task::(prepare) Task=`de327f1b-eb84-458f-9d22-00e658170711`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:23,320::task::595::TaskManager.Task::(_updateState) Task=`de327f1b-eb84-458f-9d22-00e658170711`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:23,322::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:23,325::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:23,327::task::990::TaskManager.Task::(_decref) Task=`de327f1b-eb84-458f-9d22-00e658170711`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:24,338::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:24,340::task::595::TaskManager.Task::(_updateState) Task=`674c1648-f821-4d50-a25b-a5f66d4680a2`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:24,340::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:24,341::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:24,341::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:24,341::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:24,342::task::1185::TaskManager.Task::(prepare) Task=`674c1648-f821-4d50-a25b-a5f66d4680a2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:24,342::task::595::TaskManager.Task::(_updateState) Task=`674c1648-f821-4d50-a25b-a5f66d4680a2`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:24,342::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:24,343::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:24,343::task::990::TaskManager.Task::(_decref) Task=`674c1648-f821-4d50-a25b-a5f66d4680a2`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:25,348::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:25,348::task::595::TaskManager.Task::(_updateState) Task=`c6b4f47f-ab4b-477d-a37c-c84055396e01`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:25,349::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:25,349::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:25,349::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:25,350::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:25,350::task::1185::TaskManager.Task::(prepare) Task=`c6b4f47f-ab4b-477d-a37c-c84055396e01`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:25,350::task::595::TaskManager.Task::(_updateState) Task=`c6b4f47f-ab4b-477d-a37c-c84055396e01`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:25,351::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:25,351::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:25,351::task::990::TaskManager.Task::(_decref) Task=`c6b4f47f-ab4b-477d-a37c-c84055396e01`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:26,356::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:26,395::task::595::TaskManager.Task::(_updateState) Task=`ce1afedb-1466-4658-b7ca-aa16c9b057e4`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:26,396::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:26,396::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:26,396::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:26,397::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:26,397::task::1185::TaskManager.Task::(prepare) Task=`ce1afedb-1466-4658-b7ca-aa16c9b057e4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:26,397::task::595::TaskManager.Task::(_updateState) Task=`ce1afedb-1466-4658-b7ca-aa16c9b057e4`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:26,398::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:26,398::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:26,398::task::990::TaskManager.Task::(_decref) Task=`ce1afedb-1466-4658-b7ca-aa16c9b057e4`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:27,403::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:27,404::task::595::TaskManager.Task::(_updateState) Task=`c299b807-2276-477a-8b11-3786dff393c7`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:27,405::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:27,405::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:27,405::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:27,406::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:27,406::task::1185::TaskManager.Task::(prepare) Task=`c299b807-2276-477a-8b11-3786dff393c7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:27,406::task::595::TaskManager.Task::(_updateState) Task=`c299b807-2276-477a-8b11-3786dff393c7`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:27,406::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:27,407::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:27,407::task::990::TaskManager.Task::(_decref) Task=`c299b807-2276-477a-8b11-3786dff393c7`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:28,411::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:28,412::task::595::TaskManager.Task::(_updateState) Task=`d142cf48-6c32-4efe-a48a-e9e4907018be`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:28,413::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:28,413::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:28,413::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:28,414::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:28,414::task::1185::TaskManager.Task::(prepare) Task=`d142cf48-6c32-4efe-a48a-e9e4907018be`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:28,414::task::595::TaskManager.Task::(_updateState) Task=`d142cf48-6c32-4efe-a48a-e9e4907018be`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:28,415::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:28,415::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:28,415::task::990::TaskManager.Task::(_decref) Task=`d142cf48-6c32-4efe-a48a-e9e4907018be`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:29,421::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:29,423::task::595::TaskManager.Task::(_updateState) Task=`5c6cc6cc-2c7f-4370-8e67-2b168b87deab`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:29,424::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:29,425::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:29,425::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:29,426::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:29,427::task::1185::TaskManager.Task::(prepare) Task=`5c6cc6cc-2c7f-4370-8e67-2b168b87deab`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:29,427::task::595::TaskManager.Task::(_updateState) Task=`5c6cc6cc-2c7f-4370-8e67-2b168b87deab`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:29,428::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:29,428::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:29,429::task::990::TaskManager.Task::(_decref) Task=`5c6cc6cc-2c7f-4370-8e67-2b168b87deab`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:30,445::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:30,445::task::595::TaskManager.Task::(_updateState) Task=`0debfd02-50c6-41ae-878d-07c4726f4cbb`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:30,446::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:30,446::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:30,446::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:30,447::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:30,447::task::1185::TaskManager.Task::(prepare) Task=`0debfd02-50c6-41ae-878d-07c4726f4cbb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:30,447::task::595::TaskManager.Task::(_updateState) Task=`0debfd02-50c6-41ae-878d-07c4726f4cbb`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:30,448::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:30,448::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:30,448::task::990::TaskManager.Task::(_decref) Task=`0debfd02-50c6-41ae-878d-07c4726f4cbb`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:31,453::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:31,456::task::595::TaskManager.Task::(_updateState) Task=`04ef7eae-1298-4a0b-a2c5-58bb9682b4fa`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:31,457::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:31,457::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:31,458::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:31,458::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:31,458::task::1185::TaskManager.Task::(prepare) Task=`04ef7eae-1298-4a0b-a2c5-58bb9682b4fa`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:31,459::task::595::TaskManager.Task::(_updateState) Task=`04ef7eae-1298-4a0b-a2c5-58bb9682b4fa`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:31,459::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:31,460::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:31,460::task::990::TaskManager.Task::(_decref) Task=`04ef7eae-1298-4a0b-a2c5-58bb9682b4fa`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:32,464::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:32,465::task::595::TaskManager.Task::(_updateState) Task=`17d3c09b-9f71-407b-bb74-1f701f6f416b`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:32,465::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:32,465::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:32,466::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:32,466::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:32,466::task::1185::TaskManager.Task::(prepare) Task=`17d3c09b-9f71-407b-bb74-1f701f6f416b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:32,467::task::595::TaskManager.Task::(_updateState) Task=`17d3c09b-9f71-407b-bb74-1f701f6f416b`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:32,467::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:32,467::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:32,468::task::990::TaskManager.Task::(_decref) Task=`17d3c09b-9f71-407b-bb74-1f701f6f416b`::ref 0 aborting False
>Thread-65::DEBUG::2014-04-11 22:17:33,112::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:17:33,142::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000448966 s, 1.7 MB/s\n'; <rc> = 0
>Thread-28::DEBUG::2014-04-11 22:17:33,472::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:33,473::task::595::TaskManager.Task::(_updateState) Task=`7835a927-a67b-472b-96e0-8e8c4a3d9b3e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:33,474::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:33,475::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:33,476::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:33,476::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:33,477::task::1185::TaskManager.Task::(prepare) Task=`7835a927-a67b-472b-96e0-8e8c4a3d9b3e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:33,477::task::595::TaskManager.Task::(_updateState) Task=`7835a927-a67b-472b-96e0-8e8c4a3d9b3e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:33,478::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:33,478::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:33,478::task::990::TaskManager.Task::(_decref) Task=`7835a927-a67b-472b-96e0-8e8c4a3d9b3e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:34,485::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:34,487::task::595::TaskManager.Task::(_updateState) Task=`3bbe0ca8-277e-4853-bcf9-26051b50bab3`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:34,488::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:34,489::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:34,489::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:34,490::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:34,491::task::1185::TaskManager.Task::(prepare) Task=`3bbe0ca8-277e-4853-bcf9-26051b50bab3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:34,491::task::595::TaskManager.Task::(_updateState) Task=`3bbe0ca8-277e-4853-bcf9-26051b50bab3`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:34,492::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:34,495::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:34,496::task::990::TaskManager.Task::(_decref) Task=`3bbe0ca8-277e-4853-bcf9-26051b50bab3`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:35,501::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:35,503::task::595::TaskManager.Task::(_updateState) Task=`f65d565e-0240-4491-b524-4094c1e7ae9c`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:35,504::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:35,505::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:35,505::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:35,508::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:35,509::task::1185::TaskManager.Task::(prepare) Task=`f65d565e-0240-4491-b524-4094c1e7ae9c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:35,510::task::595::TaskManager.Task::(_updateState) Task=`f65d565e-0240-4491-b524-4094c1e7ae9c`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:35,510::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:35,511::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:35,511::task::990::TaskManager.Task::(_decref) Task=`f65d565e-0240-4491-b524-4094c1e7ae9c`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:36,517::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:36,520::task::595::TaskManager.Task::(_updateState) Task=`29db3d1c-1d78-4250-9073-3aa00b58e071`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:36,520::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:36,521::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:36,521::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:36,521::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:36,522::task::1185::TaskManager.Task::(prepare) Task=`29db3d1c-1d78-4250-9073-3aa00b58e071`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:36,522::task::595::TaskManager.Task::(_updateState) Task=`29db3d1c-1d78-4250-9073-3aa00b58e071`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:36,522::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:36,522::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:36,523::task::990::TaskManager.Task::(_decref) Task=`29db3d1c-1d78-4250-9073-3aa00b58e071`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:37,527::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:37,528::task::595::TaskManager.Task::(_updateState) Task=`2c45668b-7952-40e0-a1d3-5440c3b84212`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:37,528::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:37,528::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:37,529::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:37,529::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:37,529::task::1185::TaskManager.Task::(prepare) Task=`2c45668b-7952-40e0-a1d3-5440c3b84212`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:37,530::task::595::TaskManager.Task::(_updateState) Task=`2c45668b-7952-40e0-a1d3-5440c3b84212`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:37,530::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:37,530::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:37,531::task::990::TaskManager.Task::(_decref) Task=`2c45668b-7952-40e0-a1d3-5440c3b84212`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:38,535::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:38,536::task::595::TaskManager.Task::(_updateState) Task=`0ff7754a-3578-4837-a642-61f61033ca96`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:38,536::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:38,536::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:38,536::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:38,537::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:38,537::task::1185::TaskManager.Task::(prepare) Task=`0ff7754a-3578-4837-a642-61f61033ca96`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:38,538::task::595::TaskManager.Task::(_updateState) Task=`0ff7754a-3578-4837-a642-61f61033ca96`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:38,538::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:38,538::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:38,539::task::990::TaskManager.Task::(_decref) Task=`0ff7754a-3578-4837-a642-61f61033ca96`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:39,545::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:39,547::task::595::TaskManager.Task::(_updateState) Task=`7f1b927a-ff72-4a51-938e-4fc8a7d58a26`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:39,548::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:39,549::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:39,549::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:39,549::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:39,550::task::1185::TaskManager.Task::(prepare) Task=`7f1b927a-ff72-4a51-938e-4fc8a7d58a26`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:39,550::task::595::TaskManager.Task::(_updateState) Task=`7f1b927a-ff72-4a51-938e-4fc8a7d58a26`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:39,551::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:39,551::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:39,552::task::990::TaskManager.Task::(_decref) Task=`7f1b927a-ff72-4a51-938e-4fc8a7d58a26`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:40,557::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:40,559::task::595::TaskManager.Task::(_updateState) Task=`f9930b68-a94f-4385-a1ed-054a982a990a`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:40,559::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:40,560::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:40,561::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:40,561::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:40,561::task::1185::TaskManager.Task::(prepare) Task=`f9930b68-a94f-4385-a1ed-054a982a990a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:40,563::task::595::TaskManager.Task::(_updateState) Task=`f9930b68-a94f-4385-a1ed-054a982a990a`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:40,564::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:40,564::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:40,564::task::990::TaskManager.Task::(_decref) Task=`f9930b68-a94f-4385-a1ed-054a982a990a`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:41,570::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:41,588::task::595::TaskManager.Task::(_updateState) Task=`d7a78e22-90c8-4c81-9e98-2392f1b5d495`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:41,589::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:41,590::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:41,590::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:41,590::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:41,591::task::1185::TaskManager.Task::(prepare) Task=`d7a78e22-90c8-4c81-9e98-2392f1b5d495`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:41,592::task::595::TaskManager.Task::(_updateState) Task=`d7a78e22-90c8-4c81-9e98-2392f1b5d495`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:41,592::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:41,592::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:41,593::task::990::TaskManager.Task::(_decref) Task=`d7a78e22-90c8-4c81-9e98-2392f1b5d495`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:42,598::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:42,599::task::595::TaskManager.Task::(_updateState) Task=`e119d0a6-392e-44c2-a49e-3794a86acb13`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:42,599::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:42,599::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:42,600::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:42,600::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:42,601::task::1185::TaskManager.Task::(prepare) Task=`e119d0a6-392e-44c2-a49e-3794a86acb13`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:42,601::task::595::TaskManager.Task::(_updateState) Task=`e119d0a6-392e-44c2-a49e-3794a86acb13`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:42,602::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:42,602::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:42,603::task::990::TaskManager.Task::(_decref) Task=`e119d0a6-392e-44c2-a49e-3794a86acb13`::ref 0 aborting False
>Thread-65::DEBUG::2014-04-11 22:17:43,175::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-65::DEBUG::2014-04-11 22:17:43,205::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000400997 s, 1.9 MB/s\n'; <rc> = 0
>Thread-28::DEBUG::2014-04-11 22:17:43,609::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:43,610::task::595::TaskManager.Task::(_updateState) Task=`71d18ecc-9964-4b89-b280-d5572d73efb8`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:43,611::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:43,611::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:43,612::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:43,612::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:43,612::task::1185::TaskManager.Task::(prepare) Task=`71d18ecc-9964-4b89-b280-d5572d73efb8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:43,614::task::595::TaskManager.Task::(_updateState) Task=`71d18ecc-9964-4b89-b280-d5572d73efb8`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:43,616::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:43,617::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:43,619::task::990::TaskManager.Task::(_decref) Task=`71d18ecc-9964-4b89-b280-d5572d73efb8`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:44,629::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:44,631::task::595::TaskManager.Task::(_updateState) Task=`42e7bbb5-9579-4976-a718-9014192c2ac0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:44,632::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:44,633::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:44,634::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:44,634::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:44,637::task::1185::TaskManager.Task::(prepare) Task=`42e7bbb5-9579-4976-a718-9014192c2ac0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:44,639::task::595::TaskManager.Task::(_updateState) Task=`42e7bbb5-9579-4976-a718-9014192c2ac0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:44,640::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:44,641::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:44,642::task::990::TaskManager.Task::(_decref) Task=`42e7bbb5-9579-4976-a718-9014192c2ac0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:45,652::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:45,653::task::595::TaskManager.Task::(_updateState) Task=`95014451-74ca-492c-8a63-895998fbcb9b`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:45,654::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:45,654::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:45,654::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:45,655::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:45,655::task::1185::TaskManager.Task::(prepare) Task=`95014451-74ca-492c-8a63-895998fbcb9b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:45,656::task::595::TaskManager.Task::(_updateState) Task=`95014451-74ca-492c-8a63-895998fbcb9b`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:45,656::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:45,656::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:45,657::task::990::TaskManager.Task::(_decref) Task=`95014451-74ca-492c-8a63-895998fbcb9b`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:46,661::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:46,680::task::595::TaskManager.Task::(_updateState) Task=`664a10a5-edf1-40c2-9cea-372a1ba71b27`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:46,681::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:46,681::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:46,682::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:46,682::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:46,682::task::1185::TaskManager.Task::(prepare) Task=`664a10a5-edf1-40c2-9cea-372a1ba71b27`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:46,683::task::595::TaskManager.Task::(_updateState) Task=`664a10a5-edf1-40c2-9cea-372a1ba71b27`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:46,683::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:46,684::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:46,684::task::990::TaskManager.Task::(_decref) Task=`664a10a5-edf1-40c2-9cea-372a1ba71b27`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:47,689::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:47,690::task::595::TaskManager.Task::(_updateState) Task=`450b0a22-f873-41c2-90b4-a908efdc52fd`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:47,691::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:47,691::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:47,691::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:47,692::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:47,692::task::1185::TaskManager.Task::(prepare) Task=`450b0a22-f873-41c2-90b4-a908efdc52fd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:47,692::task::595::TaskManager.Task::(_updateState) Task=`450b0a22-f873-41c2-90b4-a908efdc52fd`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:47,693::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:47,693::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:47,694::task::990::TaskManager.Task::(_decref) Task=`450b0a22-f873-41c2-90b4-a908efdc52fd`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:48,698::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:48,699::task::595::TaskManager.Task::(_updateState) Task=`1dfa1d5d-14f4-440c-b366-9a35c51e1e67`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:48,700::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:48,700::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:48,700::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:48,701::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:48,701::task::1185::TaskManager.Task::(prepare) Task=`1dfa1d5d-14f4-440c-b366-9a35c51e1e67`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:48,702::task::595::TaskManager.Task::(_updateState) Task=`1dfa1d5d-14f4-440c-b366-9a35c51e1e67`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:48,702::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:48,702::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:48,703::task::990::TaskManager.Task::(_decref) Task=`1dfa1d5d-14f4-440c-b366-9a35c51e1e67`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:49,716::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:49,731::task::595::TaskManager.Task::(_updateState) Task=`dca7c7c0-08c6-492d-b3c8-d71901092327`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:49,738::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:49,752::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:49,754::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:49,754::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:49,755::task::1185::TaskManager.Task::(prepare) Task=`dca7c7c0-08c6-492d-b3c8-d71901092327`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:49,759::task::595::TaskManager.Task::(_updateState) Task=`dca7c7c0-08c6-492d-b3c8-d71901092327`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:49,765::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:49,767::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:49,772::task::990::TaskManager.Task::(_decref) Task=`dca7c7c0-08c6-492d-b3c8-d71901092327`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:50,788::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:50,789::task::595::TaskManager.Task::(_updateState) Task=`9347db3a-c612-41b9-a49b-0974340cbc38`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:50,789::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:50,790::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:50,791::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:50,791::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:50,792::task::1185::TaskManager.Task::(prepare) Task=`9347db3a-c612-41b9-a49b-0974340cbc38`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:50,793::task::595::TaskManager.Task::(_updateState) Task=`9347db3a-c612-41b9-a49b-0974340cbc38`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:50,794::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:50,794::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:50,794::task::990::TaskManager.Task::(_decref) Task=`9347db3a-c612-41b9-a49b-0974340cbc38`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:51,800::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:51,802::task::595::TaskManager.Task::(_updateState) Task=`09400b28-4407-453c-ab88-cd1ebe1c67c8`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:51,803::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:51,804::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:51,805::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:51,806::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:51,807::task::1185::TaskManager.Task::(prepare) Task=`09400b28-4407-453c-ab88-cd1ebe1c67c8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:51,808::task::595::TaskManager.Task::(_updateState) Task=`09400b28-4407-453c-ab88-cd1ebe1c67c8`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:51,809::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:51,811::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:51,812::task::990::TaskManager.Task::(_decref) Task=`09400b28-4407-453c-ab88-cd1ebe1c67c8`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:52,825::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:52,834::task::595::TaskManager.Task::(_updateState) Task=`26814564-c837-4e71-aae1-7b720279e01e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:52,842::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:52,847::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:52,853::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:52,864::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:52,870::task::1185::TaskManager.Task::(prepare) Task=`26814564-c837-4e71-aae1-7b720279e01e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:52,874::task::595::TaskManager.Task::(_updateState) Task=`26814564-c837-4e71-aae1-7b720279e01e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:52,880::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:52,887::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:52,896::task::990::TaskManager.Task::(_decref) Task=`26814564-c837-4e71-aae1-7b720279e01e`::ref 0 aborting False
>Thread-223::WARNING::2014-04-11 22:17:53,346::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-223::DEBUG::2014-04-11 22:17:53,402::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-224::WARNING::2014-04-11 22:17:53,387::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-226::WARNING::2014-04-11 22:17:53,392::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-224::DEBUG::2014-04-11 22:17:53,454::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-225::WARNING::2014-04-11 22:17:53,392::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-226::DEBUG::2014-04-11 22:17:53,485::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-225::DEBUG::2014-04-11 22:17:53,513::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-235::WARNING::2014-04-11 22:17:53,584::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-235::DEBUG::2014-04-11 22:17:53,645::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-236::WARNING::2014-04-11 22:17:53,621::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-237::WARNING::2014-04-11 22:17:53,630::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-236::DEBUG::2014-04-11 22:17:53,680::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-238::WARNING::2014-04-11 22:17:53,671::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-237::DEBUG::2014-04-11 22:17:53,701::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-241::WARNING::2014-04-11 22:17:53,699::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-238::DEBUG::2014-04-11 22:17:53,722::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-242::WARNING::2014-04-11 22:17:53,700::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-241::DEBUG::2014-04-11 22:17:53,742::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-229::ERROR::2014-04-11 22:17:53,750::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',)
>Thread-243::WARNING::2014-04-11 22:17:53,719::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-244::WARNING::2014-04-11 22:17:53,741::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-234::ERROR::2014-04-11 22:17:53,806::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 21619c8e-99ea-4813-be02-d708971e5393 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',)
>Thread-65::DEBUG::2014-04-11 22:17:53,827::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-28::DEBUG::2014-04-11 22:17:54,035::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:54,127::task::595::TaskManager.Task::(_updateState) Task=`7c58d362-4a74-4bfb-9033-57b6d1f3aff7`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:54,148::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:54,159::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:54,162::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:54,165::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:54,168::task::1185::TaskManager.Task::(prepare) Task=`7c58d362-4a74-4bfb-9033-57b6d1f3aff7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-65::DEBUG::2014-04-11 22:17:54,170::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000423611 s, 1.8 MB/s\n'; <rc> = 0
>Thread-28::DEBUG::2014-04-11 22:17:54,171::task::595::TaskManager.Task::(_updateState) Task=`7c58d362-4a74-4bfb-9033-57b6d1f3aff7`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:54,179::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:54,182::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:54,185::task::990::TaskManager.Task::(_decref) Task=`7c58d362-4a74-4bfb-9033-57b6d1f3aff7`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:55,202::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:55,205::task::595::TaskManager.Task::(_updateState) Task=`efdda0af-bfaa-4535-82ba-655b811acf87`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:55,205::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:55,206::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:55,206::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:55,208::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:55,208::task::1185::TaskManager.Task::(prepare) Task=`efdda0af-bfaa-4535-82ba-655b811acf87`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:55,209::task::595::TaskManager.Task::(_updateState) Task=`efdda0af-bfaa-4535-82ba-655b811acf87`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:55,210::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:55,210::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:55,211::task::990::TaskManager.Task::(_decref) Task=`efdda0af-bfaa-4535-82ba-655b811acf87`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:56,216::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:56,217::task::595::TaskManager.Task::(_updateState) Task=`86ceb74a-e6a7-4211-b20e-82beed601fc9`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:56,217::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:56,218::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:56,218::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:56,219::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:56,219::task::1185::TaskManager.Task::(prepare) Task=`86ceb74a-e6a7-4211-b20e-82beed601fc9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:56,219::task::595::TaskManager.Task::(_updateState) Task=`86ceb74a-e6a7-4211-b20e-82beed601fc9`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:56,220::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:56,220::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:56,221::task::990::TaskManager.Task::(_decref) Task=`86ceb74a-e6a7-4211-b20e-82beed601fc9`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:57,230::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:57,230::task::595::TaskManager.Task::(_updateState) Task=`4c8fd0bc-0abf-4bac-85a2-a101deb9dfe5`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:57,231::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:57,231::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:57,231::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:57,232::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:57,232::task::1185::TaskManager.Task::(prepare) Task=`4c8fd0bc-0abf-4bac-85a2-a101deb9dfe5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:57,232::task::595::TaskManager.Task::(_updateState) Task=`4c8fd0bc-0abf-4bac-85a2-a101deb9dfe5`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:57,233::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:57,233::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:57,234::task::990::TaskManager.Task::(_decref) Task=`4c8fd0bc-0abf-4bac-85a2-a101deb9dfe5`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:58,239::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:58,239::task::595::TaskManager.Task::(_updateState) Task=`def69519-df72-46d4-81ce-9bef3b98c82e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:58,240::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:58,240::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:58,241::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:58,241::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:58,241::task::1185::TaskManager.Task::(prepare) Task=`def69519-df72-46d4-81ce-9bef3b98c82e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:58,242::task::595::TaskManager.Task::(_updateState) Task=`def69519-df72-46d4-81ce-9bef3b98c82e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:58,242::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:58,242::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:58,243::task::990::TaskManager.Task::(_decref) Task=`def69519-df72-46d4-81ce-9bef3b98c82e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:17:59,248::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:17:59,249::task::595::TaskManager.Task::(_updateState) Task=`e8286468-c220-4948-aac7-6d92bfcefc72`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:17:59,250::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:17:59,250::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:17:59,251::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:17:59,252::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:59,252::task::1185::TaskManager.Task::(prepare) Task=`e8286468-c220-4948-aac7-6d92bfcefc72`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:17:59,253::task::595::TaskManager.Task::(_updateState) Task=`e8286468-c220-4948-aac7-6d92bfcefc72`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:17:59,254::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:17:59,254::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:17:59,254::task::990::TaskManager.Task::(_decref) Task=`e8286468-c220-4948-aac7-6d92bfcefc72`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:00,263::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:00,264::task::595::TaskManager.Task::(_updateState) Task=`0bd681a3-ed3f-48d1-826d-4789a60cca00`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:00,264::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:00,265::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:00,265::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:00,266::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:00,266::task::1185::TaskManager.Task::(prepare) Task=`0bd681a3-ed3f-48d1-826d-4789a60cca00`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:00,266::task::595::TaskManager.Task::(_updateState) Task=`0bd681a3-ed3f-48d1-826d-4789a60cca00`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:00,267::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:00,267::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:00,267::task::990::TaskManager.Task::(_decref) Task=`0bd681a3-ed3f-48d1-826d-4789a60cca00`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:01,274::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:01,276::task::595::TaskManager.Task::(_updateState) Task=`cbe2f521-5186-4a8e-ad60-4ad16174bf0f`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:01,276::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:01,277::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:01,277::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:01,278::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:01,278::task::1185::TaskManager.Task::(prepare) Task=`cbe2f521-5186-4a8e-ad60-4ad16174bf0f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:01,278::task::595::TaskManager.Task::(_updateState) Task=`cbe2f521-5186-4a8e-ad60-4ad16174bf0f`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:01,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:01,279::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:01,279::task::990::TaskManager.Task::(_decref) Task=`cbe2f521-5186-4a8e-ad60-4ad16174bf0f`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:02,283::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:02,296::task::595::TaskManager.Task::(_updateState) Task=`39c621cd-fd99-4044-a32c-4a8eed46546c`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:02,297::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:02,298::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:02,298::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:02,298::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:02,299::task::1185::TaskManager.Task::(prepare) Task=`39c621cd-fd99-4044-a32c-4a8eed46546c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:02,299::task::595::TaskManager.Task::(_updateState) Task=`39c621cd-fd99-4044-a32c-4a8eed46546c`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:02,300::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:02,300::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:02,301::task::990::TaskManager.Task::(_decref) Task=`39c621cd-fd99-4044-a32c-4a8eed46546c`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:03,305::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:03,306::task::595::TaskManager.Task::(_updateState) Task=`ab9ef7c4-262a-4e10-ac6f-11cdc9e1a8c3`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:03,307::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:03,307::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:03,307::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:03,308::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:03,308::task::1185::TaskManager.Task::(prepare) Task=`ab9ef7c4-262a-4e10-ac6f-11cdc9e1a8c3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:03,309::task::595::TaskManager.Task::(_updateState) Task=`ab9ef7c4-262a-4e10-ac6f-11cdc9e1a8c3`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:03,309::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:03,309::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:03,310::task::990::TaskManager.Task::(_decref) Task=`ab9ef7c4-262a-4e10-ac6f-11cdc9e1a8c3`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:04,315::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:04,323::task::595::TaskManager.Task::(_updateState) Task=`c0541a3c-16cb-41aa-832a-2142f42e4aa8`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:04,324::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:04,325::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:04,325::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:04,326::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:04,326::task::1185::TaskManager.Task::(prepare) Task=`c0541a3c-16cb-41aa-832a-2142f42e4aa8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:04,330::task::595::TaskManager.Task::(_updateState) Task=`c0541a3c-16cb-41aa-832a-2142f42e4aa8`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:04,331::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:04,331::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:04,332::task::990::TaskManager.Task::(_decref) Task=`c0541a3c-16cb-41aa-832a-2142f42e4aa8`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:05,339::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:05,342::task::595::TaskManager.Task::(_updateState) Task=`5111a9d0-a9da-4e00-82d1-607535153de7`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:05,342::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:05,343::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:05,343::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:05,347::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:05,349::task::1185::TaskManager.Task::(prepare) Task=`5111a9d0-a9da-4e00-82d1-607535153de7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:05,349::task::595::TaskManager.Task::(_updateState) Task=`5111a9d0-a9da-4e00-82d1-607535153de7`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:05,349::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:05,350::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:05,354::task::990::TaskManager.Task::(_decref) Task=`5111a9d0-a9da-4e00-82d1-607535153de7`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:06,360::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:06,361::task::595::TaskManager.Task::(_updateState) Task=`314a9b83-4d11-4b5a-bc83-40a33dc95327`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:06,361::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:06,362::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:06,362::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:06,363::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:06,363::task::1185::TaskManager.Task::(prepare) Task=`314a9b83-4d11-4b5a-bc83-40a33dc95327`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:06,364::task::595::TaskManager.Task::(_updateState) Task=`314a9b83-4d11-4b5a-bc83-40a33dc95327`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:06,364::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:06,365::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:06,365::task::990::TaskManager.Task::(_decref) Task=`314a9b83-4d11-4b5a-bc83-40a33dc95327`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:07,369::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:07,371::task::595::TaskManager.Task::(_updateState) Task=`b2b03a9a-0160-467d-a438-7cdff90b1e70`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:07,371::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:07,371::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:07,372::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:07,372::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:07,373::task::1185::TaskManager.Task::(prepare) Task=`b2b03a9a-0160-467d-a438-7cdff90b1e70`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:07,373::task::595::TaskManager.Task::(_updateState) Task=`b2b03a9a-0160-467d-a438-7cdff90b1e70`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:07,374::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:07,374::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:07,375::task::990::TaskManager.Task::(_decref) Task=`b2b03a9a-0160-467d-a438-7cdff90b1e70`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:08,379::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:08,380::task::595::TaskManager.Task::(_updateState) Task=`86249225-1df5-4531-8155-e980007c71bb`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:08,381::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:08,382::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:08,382::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:08,383::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:08,383::task::1185::TaskManager.Task::(prepare) Task=`86249225-1df5-4531-8155-e980007c71bb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:08,384::task::595::TaskManager.Task::(_updateState) Task=`86249225-1df5-4531-8155-e980007c71bb`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:08,385::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:08,385::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:08,385::task::990::TaskManager.Task::(_decref) Task=`86249225-1df5-4531-8155-e980007c71bb`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:09,431::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:09,473::task::595::TaskManager.Task::(_updateState) Task=`01e9ac2c-80a4-4f23-8355-5170b4cb23da`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:09,510::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:09,536::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:09,564::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:09,595::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:09,625::task::1185::TaskManager.Task::(prepare) Task=`01e9ac2c-80a4-4f23-8355-5170b4cb23da`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:09,658::task::595::TaskManager.Task::(_updateState) Task=`01e9ac2c-80a4-4f23-8355-5170b4cb23da`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:09,688::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:09,720::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:09,758::task::990::TaskManager.Task::(_decref) Task=`01e9ac2c-80a4-4f23-8355-5170b4cb23da`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:10,930::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:10,996::task::595::TaskManager.Task::(_updateState) Task=`62e623c2-df9c-47c1-82d3-aadf0a39bf24`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:11,050::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:11,090::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:11,132::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:11,171::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:11,213::task::1185::TaskManager.Task::(prepare) Task=`62e623c2-df9c-47c1-82d3-aadf0a39bf24`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:11,257::task::595::TaskManager.Task::(_updateState) Task=`62e623c2-df9c-47c1-82d3-aadf0a39bf24`::moving from state preparing -> state finished
>Thread-247::WARNING::2014-04-11 22:18:11,234::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-28::DEBUG::2014-04-11 22:18:11,295::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-248::WARNING::2014-04-11 22:18:11,262::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-247::DEBUG::2014-04-11 22:18:11,307::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-249::WARNING::2014-04-11 22:18:11,270::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-28::DEBUG::2014-04-11 22:18:11,330::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-248::DEBUG::2014-04-11 22:18:11,341::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-250::WARNING::2014-04-11 22:18:11,277::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-249::DEBUG::2014-04-11 22:18:11,366::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-28::DEBUG::2014-04-11 22:18:11,382::task::990::TaskManager.Task::(_decref) Task=`62e623c2-df9c-47c1-82d3-aadf0a39bf24`::ref 0 aborting False
>Thread-250::DEBUG::2014-04-11 22:18:11,399::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-228::ERROR::2014-04-11 22:18:11,441::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-254::WARNING::2014-04-11 22:18:11,453::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-72::ERROR::2014-04-11 22:18:11,491::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-255::WARNING::2014-04-11 22:18:11,481::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-72::ERROR::2014-04-11 22:18:11,515::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-254::DEBUG::2014-04-11 22:18:11,518::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-256::WARNING::2014-04-11 22:18:11,495::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-255::DEBUG::2014-04-11 22:18:11,538::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-257::WARNING::2014-04-11 22:18:11,495::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-72::DEBUG::2014-04-11 22:18:11,556::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-256::DEBUG::2014-04-11 22:18:11,559::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-258::WARNING::2014-04-11 22:18:11,541::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-257::DEBUG::2014-04-11 22:18:11,575::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-259::WARNING::2014-04-11 22:18:11,542::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-258::DEBUG::2014-04-11 22:18:11,594::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-260::WARNING::2014-04-11 22:18:11,594::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-259::DEBUG::2014-04-11 22:18:11,610::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-261::WARNING::2014-04-11 22:18:11,603::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-260::DEBUG::2014-04-11 22:18:11,627::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-261::DEBUG::2014-04-11 22:18:11,638::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-230::ERROR::2014-04-11 22:18:11,647::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-265::WARNING::2014-04-11 22:18:11,636::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-232::ERROR::2014-04-11 22:18:11,656::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-72::DEBUG::2014-04-11 22:18:11,657::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None)
>Thread-74::ERROR::2014-04-11 22:18:11,670::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-265::DEBUG::2014-04-11 22:18:11,672::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-268::DEBUG::2014-04-11 22:18:11,672::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-266::WARNING::2014-04-11 22:18:11,657::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-82::ERROR::2014-04-11 22:18:11,692::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-74::ERROR::2014-04-11 22:18:11,700::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-267::WARNING::2014-04-11 22:18:11,672::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-82::ERROR::2014-04-11 22:18:11,715::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-266::DEBUG::2014-04-11 22:18:11,715::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-268::WARNING::2014-04-11 22:18:11,712::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-267::DEBUG::2014-04-11 22:18:11,723::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-74::DEBUG::2014-04-11 22:18:11,724::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-268::DEBUG::2014-04-11 22:18:11,728::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-82::DEBUG::2014-04-11 22:18:11,736::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-231::ERROR::2014-04-11 22:18:11,736::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-81::ERROR::2014-04-11 22:18:11,745::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-81::ERROR::2014-04-11 22:18:11,750::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-74::DEBUG::2014-04-11 22:18:11,757::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None)
>Thread-82::DEBUG::2014-04-11 22:18:11,758::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-81::DEBUG::2014-04-11 22:18:11,758::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-81::DEBUG::2014-04-11 22:18:11,782::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-72::DEBUG::2014-04-11 22:18:12,013::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5
>Thread-81::DEBUG::2014-04-11 22:18:12,024::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-72::WARNING::2014-04-11 22:18:12,026::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8']
>Thread-81::WARNING::2014-04-11 22:18:12,026::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-74::DEBUG::2014-04-11 22:18:12,155::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5
>Thread-82::DEBUG::2014-04-11 22:18:12,243::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-74::WARNING::2014-04-11 22:18:12,274::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce']
>Thread-82::WARNING::2014-04-11 22:18:12,274::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-82::DEBUG::2014-04-11 22:18:12,283::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:18:12,532::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:12,538::task::595::TaskManager.Task::(_updateState) Task=`92dcd58a-1b57-4dd3-9584-5fe0f05ce5f6`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:12,539::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:12,540::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:12,540::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:12,541::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:12,542::task::1185::TaskManager.Task::(prepare) Task=`92dcd58a-1b57-4dd3-9584-5fe0f05ce5f6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:12,542::task::595::TaskManager.Task::(_updateState) Task=`92dcd58a-1b57-4dd3-9584-5fe0f05ce5f6`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:12,543::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:12,546::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:12,547::task::990::TaskManager.Task::(_decref) Task=`92dcd58a-1b57-4dd3-9584-5fe0f05ce5f6`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:13,553::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:13,554::task::595::TaskManager.Task::(_updateState) Task=`dc34b025-50a2-4e91-ae1b-78a201df0490`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:13,554::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:13,554::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:13,555::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:13,555::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:13,555::task::1185::TaskManager.Task::(prepare) Task=`dc34b025-50a2-4e91-ae1b-78a201df0490`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:13,556::task::595::TaskManager.Task::(_updateState) Task=`dc34b025-50a2-4e91-ae1b-78a201df0490`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:13,556::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:13,556::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:13,557::task::990::TaskManager.Task::(_decref) Task=`dc34b025-50a2-4e91-ae1b-78a201df0490`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:14,563::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:14,565::task::595::TaskManager.Task::(_updateState) Task=`c7e15481-fb87-4420-8088-9fd841a79db0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:14,565::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:14,566::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:14,566::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:14,567::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:14,567::task::1185::TaskManager.Task::(prepare) Task=`c7e15481-fb87-4420-8088-9fd841a79db0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:14,567::task::595::TaskManager.Task::(_updateState) Task=`c7e15481-fb87-4420-8088-9fd841a79db0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:14,568::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:14,568::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:14,568::task::990::TaskManager.Task::(_decref) Task=`c7e15481-fb87-4420-8088-9fd841a79db0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:15,574::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:15,575::task::595::TaskManager.Task::(_updateState) Task=`cf8df286-c10f-45fb-b06c-dcacbdc8f0d1`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:15,575::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:15,575::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:15,575::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:15,576::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:15,576::task::1185::TaskManager.Task::(prepare) Task=`cf8df286-c10f-45fb-b06c-dcacbdc8f0d1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:15,576::task::595::TaskManager.Task::(_updateState) Task=`cf8df286-c10f-45fb-b06c-dcacbdc8f0d1`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:15,577::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:15,577::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:15,577::task::990::TaskManager.Task::(_decref) Task=`cf8df286-c10f-45fb-b06c-dcacbdc8f0d1`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:16,581::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:16,582::task::595::TaskManager.Task::(_updateState) Task=`9ef17120-c0f9-4d5a-88de-1ee52bffec2e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:16,582::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:16,582::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:16,583::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:16,583::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:16,583::task::1185::TaskManager.Task::(prepare) Task=`9ef17120-c0f9-4d5a-88de-1ee52bffec2e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:16,584::task::595::TaskManager.Task::(_updateState) Task=`9ef17120-c0f9-4d5a-88de-1ee52bffec2e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:16,584::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:16,584::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:16,584::task::990::TaskManager.Task::(_decref) Task=`9ef17120-c0f9-4d5a-88de-1ee52bffec2e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:17,588::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:17,589::task::595::TaskManager.Task::(_updateState) Task=`f7ee35f1-31c3-4247-8409-bc2aed6be144`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:17,589::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:17,590::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:17,590::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:17,590::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:17,591::task::1185::TaskManager.Task::(prepare) Task=`f7ee35f1-31c3-4247-8409-bc2aed6be144`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:17,591::task::595::TaskManager.Task::(_updateState) Task=`f7ee35f1-31c3-4247-8409-bc2aed6be144`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:17,591::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:17,591::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:17,592::task::990::TaskManager.Task::(_decref) Task=`f7ee35f1-31c3-4247-8409-bc2aed6be144`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:18,596::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:18,596::task::595::TaskManager.Task::(_updateState) Task=`345a78f4-4b1c-46ff-b67a-7865a8181ec7`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:18,597::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:18,597::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:18,597::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:18,597::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:18,598::task::1185::TaskManager.Task::(prepare) Task=`345a78f4-4b1c-46ff-b67a-7865a8181ec7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:18,598::task::595::TaskManager.Task::(_updateState) Task=`345a78f4-4b1c-46ff-b67a-7865a8181ec7`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:18,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:18,599::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:18,599::task::990::TaskManager.Task::(_decref) Task=`345a78f4-4b1c-46ff-b67a-7865a8181ec7`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:19,604::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:19,605::task::595::TaskManager.Task::(_updateState) Task=`03158728-e18b-4576-88e8-a723cb9da9c0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:19,606::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:19,606::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:19,606::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:19,607::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:19,607::task::1185::TaskManager.Task::(prepare) Task=`03158728-e18b-4576-88e8-a723cb9da9c0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:19,608::task::595::TaskManager.Task::(_updateState) Task=`03158728-e18b-4576-88e8-a723cb9da9c0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:19,609::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:19,609::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:19,609::task::990::TaskManager.Task::(_decref) Task=`03158728-e18b-4576-88e8-a723cb9da9c0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:20,615::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:20,616::task::595::TaskManager.Task::(_updateState) Task=`ffeacdd2-0f93-450b-8472-7523089bb602`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:20,616::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:20,617::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:20,617::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:20,617::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:20,618::task::1185::TaskManager.Task::(prepare) Task=`ffeacdd2-0f93-450b-8472-7523089bb602`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:20,618::task::595::TaskManager.Task::(_updateState) Task=`ffeacdd2-0f93-450b-8472-7523089bb602`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:20,618::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:20,619::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:20,619::task::990::TaskManager.Task::(_decref) Task=`ffeacdd2-0f93-450b-8472-7523089bb602`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:21,623::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:21,623::task::595::TaskManager.Task::(_updateState) Task=`cc944da9-0bb7-45b5-a817-3c3e7bb3fbe5`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:21,624::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:21,624::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:21,624::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:21,625::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:21,625::task::1185::TaskManager.Task::(prepare) Task=`cc944da9-0bb7-45b5-a817-3c3e7bb3fbe5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:21,625::task::595::TaskManager.Task::(_updateState) Task=`cc944da9-0bb7-45b5-a817-3c3e7bb3fbe5`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:21,626::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:21,626::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:21,626::task::990::TaskManager.Task::(_decref) Task=`cc944da9-0bb7-45b5-a817-3c3e7bb3fbe5`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:22,630::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:22,632::task::595::TaskManager.Task::(_updateState) Task=`6433e4a0-66f7-457a-ba30-a06889966dde`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:22,633::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:22,633::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:22,633::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:22,634::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:22,634::task::1185::TaskManager.Task::(prepare) Task=`6433e4a0-66f7-457a-ba30-a06889966dde`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:22,634::task::595::TaskManager.Task::(_updateState) Task=`6433e4a0-66f7-457a-ba30-a06889966dde`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:22,635::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:22,635::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:22,635::task::990::TaskManager.Task::(_decref) Task=`6433e4a0-66f7-457a-ba30-a06889966dde`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:23,639::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:23,640::task::595::TaskManager.Task::(_updateState) Task=`5d650740-2c0e-4fde-91fb-8e7f845fc0c4`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:23,640::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:23,641::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:23,641::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:23,641::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:23,642::task::1185::TaskManager.Task::(prepare) Task=`5d650740-2c0e-4fde-91fb-8e7f845fc0c4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:23,642::task::595::TaskManager.Task::(_updateState) Task=`5d650740-2c0e-4fde-91fb-8e7f845fc0c4`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:23,642::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:23,643::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:23,643::task::990::TaskManager.Task::(_decref) Task=`5d650740-2c0e-4fde-91fb-8e7f845fc0c4`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:24,648::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:24,650::task::595::TaskManager.Task::(_updateState) Task=`d89c7e60-68e5-4e1e-93ba-e383421d3b85`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:24,650::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:24,651::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:24,651::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:24,652::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:24,652::task::1185::TaskManager.Task::(prepare) Task=`d89c7e60-68e5-4e1e-93ba-e383421d3b85`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:24,653::task::595::TaskManager.Task::(_updateState) Task=`d89c7e60-68e5-4e1e-93ba-e383421d3b85`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:24,653::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:24,653::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:24,654::task::990::TaskManager.Task::(_decref) Task=`d89c7e60-68e5-4e1e-93ba-e383421d3b85`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:25,660::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:25,661::task::595::TaskManager.Task::(_updateState) Task=`de9bd4d3-cd3a-41df-9b79-b99d90239ac4`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:25,662::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:25,662::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:25,663::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:25,663::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:25,663::task::1185::TaskManager.Task::(prepare) Task=`de9bd4d3-cd3a-41df-9b79-b99d90239ac4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:25,664::task::595::TaskManager.Task::(_updateState) Task=`de9bd4d3-cd3a-41df-9b79-b99d90239ac4`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:25,664::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:25,665::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:25,665::task::990::TaskManager.Task::(_decref) Task=`de9bd4d3-cd3a-41df-9b79-b99d90239ac4`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:26,670::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:26,671::task::595::TaskManager.Task::(_updateState) Task=`c19a6aee-2c3c-42c4-8e24-0143a0cbd0b3`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:26,671::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:26,671::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:26,672::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:26,673::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:26,673::task::1185::TaskManager.Task::(prepare) Task=`c19a6aee-2c3c-42c4-8e24-0143a0cbd0b3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:26,674::task::595::TaskManager.Task::(_updateState) Task=`c19a6aee-2c3c-42c4-8e24-0143a0cbd0b3`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:26,674::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:26,674::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:26,675::task::990::TaskManager.Task::(_decref) Task=`c19a6aee-2c3c-42c4-8e24-0143a0cbd0b3`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:27,679::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:27,683::task::595::TaskManager.Task::(_updateState) Task=`87596d5b-080f-4e8a-9bb8-03a4cb247c10`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:27,684::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:27,684::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:27,684::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:27,685::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:27,685::task::1185::TaskManager.Task::(prepare) Task=`87596d5b-080f-4e8a-9bb8-03a4cb247c10`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:27,686::task::595::TaskManager.Task::(_updateState) Task=`87596d5b-080f-4e8a-9bb8-03a4cb247c10`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:27,686::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:27,686::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:27,686::task::990::TaskManager.Task::(_decref) Task=`87596d5b-080f-4e8a-9bb8-03a4cb247c10`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:28,691::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:28,691::task::595::TaskManager.Task::(_updateState) Task=`10085ea4-4aeb-4999-acc2-2c37f9851039`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:28,692::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:28,692::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:28,692::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:28,693::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:28,693::task::1185::TaskManager.Task::(prepare) Task=`10085ea4-4aeb-4999-acc2-2c37f9851039`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:28,693::task::595::TaskManager.Task::(_updateState) Task=`10085ea4-4aeb-4999-acc2-2c37f9851039`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:28,694::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:28,694::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:28,694::task::990::TaskManager.Task::(_decref) Task=`10085ea4-4aeb-4999-acc2-2c37f9851039`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:29,700::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:29,702::task::595::TaskManager.Task::(_updateState) Task=`3d465e22-26a1-4849-bac9-327c21f85c8f`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:29,704::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:29,704::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:29,704::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:29,705::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:29,705::task::1185::TaskManager.Task::(prepare) Task=`3d465e22-26a1-4849-bac9-327c21f85c8f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:29,706::task::595::TaskManager.Task::(_updateState) Task=`3d465e22-26a1-4849-bac9-327c21f85c8f`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:29,706::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:29,707::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:29,707::task::990::TaskManager.Task::(_decref) Task=`3d465e22-26a1-4849-bac9-327c21f85c8f`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:30,712::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:30,713::task::595::TaskManager.Task::(_updateState) Task=`725f09ab-e5a5-4c5c-8b2d-3a3e1805b2a1`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:30,713::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:30,714::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:30,714::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:30,714::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:30,715::task::1185::TaskManager.Task::(prepare) Task=`725f09ab-e5a5-4c5c-8b2d-3a3e1805b2a1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:30,715::task::595::TaskManager.Task::(_updateState) Task=`725f09ab-e5a5-4c5c-8b2d-3a3e1805b2a1`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:30,715::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:30,716::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:30,716::task::990::TaskManager.Task::(_decref) Task=`725f09ab-e5a5-4c5c-8b2d-3a3e1805b2a1`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:31,720::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:31,721::task::595::TaskManager.Task::(_updateState) Task=`6728b7bf-04b5-4de1-8b82-77c49883b322`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:31,721::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:31,722::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:31,722::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:31,722::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:31,723::task::1185::TaskManager.Task::(prepare) Task=`6728b7bf-04b5-4de1-8b82-77c49883b322`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:31,723::task::595::TaskManager.Task::(_updateState) Task=`6728b7bf-04b5-4de1-8b82-77c49883b322`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:31,723::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:31,724::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:31,724::task::990::TaskManager.Task::(_decref) Task=`6728b7bf-04b5-4de1-8b82-77c49883b322`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:32,728::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:32,731::task::595::TaskManager.Task::(_updateState) Task=`b889178c-db88-4671-91d2-ac521cc43147`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:32,731::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:32,732::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:32,732::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:32,732::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:32,732::task::1185::TaskManager.Task::(prepare) Task=`b889178c-db88-4671-91d2-ac521cc43147`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:32,733::task::595::TaskManager.Task::(_updateState) Task=`b889178c-db88-4671-91d2-ac521cc43147`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:32,733::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:32,733::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:32,734::task::990::TaskManager.Task::(_decref) Task=`b889178c-db88-4671-91d2-ac521cc43147`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:33,738::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:33,739::task::595::TaskManager.Task::(_updateState) Task=`a10248c0-e8bc-4dc9-883b-e79759e087da`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:33,739::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:33,739::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:33,740::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:33,740::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:33,740::task::1185::TaskManager.Task::(prepare) Task=`a10248c0-e8bc-4dc9-883b-e79759e087da`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:33,741::task::595::TaskManager.Task::(_updateState) Task=`a10248c0-e8bc-4dc9-883b-e79759e087da`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:33,741::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:33,741::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:33,742::task::990::TaskManager.Task::(_decref) Task=`a10248c0-e8bc-4dc9-883b-e79759e087da`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:34,747::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:34,748::task::595::TaskManager.Task::(_updateState) Task=`b7523cd7-d7e1-49e3-982f-96c6c2e50111`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:34,749::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:34,749::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:34,750::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:34,750::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:34,751::task::1185::TaskManager.Task::(prepare) Task=`b7523cd7-d7e1-49e3-982f-96c6c2e50111`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:34,752::task::595::TaskManager.Task::(_updateState) Task=`b7523cd7-d7e1-49e3-982f-96c6c2e50111`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:34,753::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:34,754::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:34,755::task::990::TaskManager.Task::(_decref) Task=`b7523cd7-d7e1-49e3-982f-96c6c2e50111`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:35,761::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:35,762::task::595::TaskManager.Task::(_updateState) Task=`ff79f9d6-a9a6-401f-bd99-8f709c37209e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:35,763::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:35,763::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:35,764::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:35,765::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:35,765::task::1185::TaskManager.Task::(prepare) Task=`ff79f9d6-a9a6-401f-bd99-8f709c37209e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:35,765::task::595::TaskManager.Task::(_updateState) Task=`ff79f9d6-a9a6-401f-bd99-8f709c37209e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:35,766::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:35,766::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:35,769::task::990::TaskManager.Task::(_decref) Task=`ff79f9d6-a9a6-401f-bd99-8f709c37209e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:36,774::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:36,774::task::595::TaskManager.Task::(_updateState) Task=`100a6adf-f783-4bb8-9249-61fe49cee090`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:36,775::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:36,775::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:36,775::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:36,775::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:36,776::task::1185::TaskManager.Task::(prepare) Task=`100a6adf-f783-4bb8-9249-61fe49cee090`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:36,776::task::595::TaskManager.Task::(_updateState) Task=`100a6adf-f783-4bb8-9249-61fe49cee090`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:36,777::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:36,777::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:36,777::task::990::TaskManager.Task::(_decref) Task=`100a6adf-f783-4bb8-9249-61fe49cee090`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:37,782::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:37,798::task::595::TaskManager.Task::(_updateState) Task=`a2a9afe1-95e7-4776-b20f-8ed0446d0f22`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:37,799::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:37,799::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:37,800::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:37,800::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:37,801::task::1185::TaskManager.Task::(prepare) Task=`a2a9afe1-95e7-4776-b20f-8ed0446d0f22`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:37,801::task::595::TaskManager.Task::(_updateState) Task=`a2a9afe1-95e7-4776-b20f-8ed0446d0f22`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:37,801::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:37,802::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:37,802::task::990::TaskManager.Task::(_decref) Task=`a2a9afe1-95e7-4776-b20f-8ed0446d0f22`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:38,806::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:38,807::task::595::TaskManager.Task::(_updateState) Task=`67fe117d-2fab-4730-8e33-f7dd0e7ae45e`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:38,808::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:38,808::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:38,808::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:38,809::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:38,809::task::1185::TaskManager.Task::(prepare) Task=`67fe117d-2fab-4730-8e33-f7dd0e7ae45e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:38,809::task::595::TaskManager.Task::(_updateState) Task=`67fe117d-2fab-4730-8e33-f7dd0e7ae45e`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:38,809::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:38,810::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:38,810::task::990::TaskManager.Task::(_decref) Task=`67fe117d-2fab-4730-8e33-f7dd0e7ae45e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:39,824::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:39,826::task::595::TaskManager.Task::(_updateState) Task=`9572a15f-cb81-436a-8468-1d8d61b38470`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:39,826::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:39,826::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:39,827::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:39,828::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:39,828::task::1185::TaskManager.Task::(prepare) Task=`9572a15f-cb81-436a-8468-1d8d61b38470`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:39,829::task::595::TaskManager.Task::(_updateState) Task=`9572a15f-cb81-436a-8468-1d8d61b38470`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:39,829::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:39,829::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:39,830::task::990::TaskManager.Task::(_decref) Task=`9572a15f-cb81-436a-8468-1d8d61b38470`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:40,837::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:40,838::task::595::TaskManager.Task::(_updateState) Task=`3df8b062-8349-477d-81a0-efeea41371b6`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:40,838::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:40,840::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:40,840::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:40,840::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:40,841::task::1185::TaskManager.Task::(prepare) Task=`3df8b062-8349-477d-81a0-efeea41371b6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:40,842::task::595::TaskManager.Task::(_updateState) Task=`3df8b062-8349-477d-81a0-efeea41371b6`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:40,843::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:40,843::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:40,844::task::990::TaskManager.Task::(_decref) Task=`3df8b062-8349-477d-81a0-efeea41371b6`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:41,848::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:41,860::task::595::TaskManager.Task::(_updateState) Task=`afa34c1b-7894-492f-aa51-2cdf6e4f4527`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:41,860::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:41,860::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:41,861::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:41,861::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:41,861::task::1185::TaskManager.Task::(prepare) Task=`afa34c1b-7894-492f-aa51-2cdf6e4f4527`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:41,862::task::595::TaskManager.Task::(_updateState) Task=`afa34c1b-7894-492f-aa51-2cdf6e4f4527`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:41,862::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:41,862::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:41,863::task::990::TaskManager.Task::(_decref) Task=`afa34c1b-7894-492f-aa51-2cdf6e4f4527`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:42,868::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:42,880::task::595::TaskManager.Task::(_updateState) Task=`add4e5cd-2924-48b0-ae51-ea58aeeb72d2`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:42,881::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:42,881::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:42,881::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:42,882::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:42,882::task::1185::TaskManager.Task::(prepare) Task=`add4e5cd-2924-48b0-ae51-ea58aeeb72d2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:42,882::task::595::TaskManager.Task::(_updateState) Task=`add4e5cd-2924-48b0-ae51-ea58aeeb72d2`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:42,883::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:42,883::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:42,883::task::990::TaskManager.Task::(_decref) Task=`add4e5cd-2924-48b0-ae51-ea58aeeb72d2`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:43,888::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:43,889::task::595::TaskManager.Task::(_updateState) Task=`8e163546-20f8-4312-a0a8-35dc5748a107`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:43,889::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:43,889::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:43,889::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:43,890::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:43,890::task::1185::TaskManager.Task::(prepare) Task=`8e163546-20f8-4312-a0a8-35dc5748a107`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:43,890::task::595::TaskManager.Task::(_updateState) Task=`8e163546-20f8-4312-a0a8-35dc5748a107`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:43,891::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:43,891::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:43,892::task::990::TaskManager.Task::(_decref) Task=`8e163546-20f8-4312-a0a8-35dc5748a107`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:44,897::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:44,898::task::595::TaskManager.Task::(_updateState) Task=`a7edda22-dfbd-4882-9020-7d549f349877`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:44,899::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:44,899::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:44,899::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:44,900::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:44,900::task::1185::TaskManager.Task::(prepare) Task=`a7edda22-dfbd-4882-9020-7d549f349877`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:44,900::task::595::TaskManager.Task::(_updateState) Task=`a7edda22-dfbd-4882-9020-7d549f349877`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:44,901::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:44,901::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:44,901::task::990::TaskManager.Task::(_decref) Task=`a7edda22-dfbd-4882-9020-7d549f349877`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:45,906::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:45,907::task::595::TaskManager.Task::(_updateState) Task=`4bc4ced6-307b-42cd-b5bb-8dece2463fb0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:45,908::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:45,908::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:45,908::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:45,909::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:45,909::task::1185::TaskManager.Task::(prepare) Task=`4bc4ced6-307b-42cd-b5bb-8dece2463fb0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:45,909::task::595::TaskManager.Task::(_updateState) Task=`4bc4ced6-307b-42cd-b5bb-8dece2463fb0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:45,910::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:45,910::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:45,910::task::990::TaskManager.Task::(_decref) Task=`4bc4ced6-307b-42cd-b5bb-8dece2463fb0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:46,915::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:46,915::task::595::TaskManager.Task::(_updateState) Task=`63dbaf1f-a8ab-4b4c-b0f1-dc66ea918d76`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:46,916::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:46,916::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:46,916::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:46,917::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:46,917::task::1185::TaskManager.Task::(prepare) Task=`63dbaf1f-a8ab-4b4c-b0f1-dc66ea918d76`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:46,918::task::595::TaskManager.Task::(_updateState) Task=`63dbaf1f-a8ab-4b4c-b0f1-dc66ea918d76`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:46,918::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:46,918::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:46,919::task::990::TaskManager.Task::(_decref) Task=`63dbaf1f-a8ab-4b4c-b0f1-dc66ea918d76`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:47,936::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:47,936::task::595::TaskManager.Task::(_updateState) Task=`786bb5f7-ebf6-4dd7-9790-d74d01c09cb0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:47,937::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:47,937::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:47,937::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:47,938::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:47,938::task::1185::TaskManager.Task::(prepare) Task=`786bb5f7-ebf6-4dd7-9790-d74d01c09cb0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:47,938::task::595::TaskManager.Task::(_updateState) Task=`786bb5f7-ebf6-4dd7-9790-d74d01c09cb0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:47,939::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:47,939::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:47,939::task::990::TaskManager.Task::(_decref) Task=`786bb5f7-ebf6-4dd7-9790-d74d01c09cb0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:48,944::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:48,944::task::595::TaskManager.Task::(_updateState) Task=`6bcecac2-9ee0-4a29-a84e-b119496eaeec`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:48,945::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:48,946::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:48,946::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:48,946::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:48,947::task::1185::TaskManager.Task::(prepare) Task=`6bcecac2-9ee0-4a29-a84e-b119496eaeec`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:48,947::task::595::TaskManager.Task::(_updateState) Task=`6bcecac2-9ee0-4a29-a84e-b119496eaeec`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:48,947::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:48,947::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:48,948::task::990::TaskManager.Task::(_decref) Task=`6bcecac2-9ee0-4a29-a84e-b119496eaeec`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:49,957::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:49,962::task::595::TaskManager.Task::(_updateState) Task=`546b30c9-e006-4f69-a209-af2d3c3041a4`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:49,964::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:49,965::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:49,966::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:49,966::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:49,968::task::1185::TaskManager.Task::(prepare) Task=`546b30c9-e006-4f69-a209-af2d3c3041a4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:49,973::task::595::TaskManager.Task::(_updateState) Task=`546b30c9-e006-4f69-a209-af2d3c3041a4`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:49,977::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:49,978::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:49,978::task::990::TaskManager.Task::(_decref) Task=`546b30c9-e006-4f69-a209-af2d3c3041a4`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:50,986::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:50,987::task::595::TaskManager.Task::(_updateState) Task=`22b79349-d6df-432f-be15-b7a961488c6a`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:50,988::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:50,989::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:50,989::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:50,990::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:50,990::task::1185::TaskManager.Task::(prepare) Task=`22b79349-d6df-432f-be15-b7a961488c6a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:50,990::task::595::TaskManager.Task::(_updateState) Task=`22b79349-d6df-432f-be15-b7a961488c6a`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:50,991::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:50,991::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:50,992::task::990::TaskManager.Task::(_decref) Task=`22b79349-d6df-432f-be15-b7a961488c6a`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:51,997::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:51,998::task::595::TaskManager.Task::(_updateState) Task=`94cbea1b-e831-4ac9-bd42-8f7a0dd89525`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:51,999::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:51,999::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:52,000::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:52,001::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:52,001::task::1185::TaskManager.Task::(prepare) Task=`94cbea1b-e831-4ac9-bd42-8f7a0dd89525`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:52,002::task::595::TaskManager.Task::(_updateState) Task=`94cbea1b-e831-4ac9-bd42-8f7a0dd89525`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:52,002::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:52,003::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:52,004::task::990::TaskManager.Task::(_decref) Task=`94cbea1b-e831-4ac9-bd42-8f7a0dd89525`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:53,010::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:53,014::task::595::TaskManager.Task::(_updateState) Task=`21fd9865-246e-4662-bff7-4c6e4e46df9a`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:53,016::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:53,019::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:53,021::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:53,028::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:53,029::task::1185::TaskManager.Task::(prepare) Task=`21fd9865-246e-4662-bff7-4c6e4e46df9a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:53,031::task::595::TaskManager.Task::(_updateState) Task=`21fd9865-246e-4662-bff7-4c6e4e46df9a`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:53,034::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:53,038::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:53,043::task::990::TaskManager.Task::(_decref) Task=`21fd9865-246e-4662-bff7-4c6e4e46df9a`::ref 0 aborting False
>Thread-271::WARNING::2014-04-11 22:18:53,850::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-272::WARNING::2014-04-11 22:18:53,852::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-271::DEBUG::2014-04-11 22:18:53,891::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-272::DEBUG::2014-04-11 22:18:53,909::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-273::WARNING::2014-04-11 22:18:53,952::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-274::WARNING::2014-04-11 22:18:53,953::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-273::DEBUG::2014-04-11 22:18:53,977::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-274::DEBUG::2014-04-11 22:18:53,990::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-73::ERROR::2014-04-11 22:18:54,012::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',)
>Thread-73::ERROR::2014-04-11 22:18:54,031::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>    return getattr(self.getRealDomain(), attrName)
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',)
>Thread-28::DEBUG::2014-04-11 22:18:54,072::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-277::WARNING::2014-04-11 22:18:54,081::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-28::DEBUG::2014-04-11 22:18:54,096::task::595::TaskManager.Task::(_updateState) Task=`448a63be-7d18-44f9-ad12-a4af4d7c550e`::moving from state init -> state preparing
>Thread-277::DEBUG::2014-04-11 22:18:54,110::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-28::INFO::2014-04-11 22:18:54,117::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:54,132::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-278::WARNING::2014-04-11 22:18:54,125::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-28::DEBUG::2014-04-11 22:18:54,142::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-279::WARNING::2014-04-11 22:18:54,131::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-278::DEBUG::2014-04-11 22:18:54,147::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-280::WARNING::2014-04-11 22:18:54,143::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-28::INFO::2014-04-11 22:18:54,154::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-279::DEBUG::2014-04-11 22:18:54,158::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-280::DEBUG::2014-04-11 22:18:54,165::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-28::DEBUG::2014-04-11 22:18:54,166::task::1185::TaskManager.Task::(prepare) Task=`448a63be-7d18-44f9-ad12-a4af4d7c550e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:54,176::task::595::TaskManager.Task::(_updateState) Task=`448a63be-7d18-44f9-ad12-a4af4d7c550e`::moving from state preparing -> state finished
>Thread-83::ERROR::2014-04-11 22:18:54,177::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 21619c8e-99ea-4813-be02-d708971e5393 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',)
>Thread-28::DEBUG::2014-04-11 22:18:54,183::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-83::ERROR::2014-04-11 22:18:54,187::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 21619c8e-99ea-4813-be02-d708971e5393 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
>    self.domain.selftest()
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>    return getattr(self.getRealDomain(), attrName)
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
>    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',)
>Thread-28::DEBUG::2014-04-11 22:18:54,189::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:54,200::task::990::TaskManager.Task::(_decref) Task=`448a63be-7d18-44f9-ad12-a4af4d7c550e`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:55,211::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:55,213::task::595::TaskManager.Task::(_updateState) Task=`bb0a5da6-ef21-4250-9a01-c468e9cd36dd`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:55,214::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:55,215::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:55,215::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:55,215::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:55,216::task::1185::TaskManager.Task::(prepare) Task=`bb0a5da6-ef21-4250-9a01-c468e9cd36dd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:55,216::task::595::TaskManager.Task::(_updateState) Task=`bb0a5da6-ef21-4250-9a01-c468e9cd36dd`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:55,217::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:55,219::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:55,219::task::990::TaskManager.Task::(_decref) Task=`bb0a5da6-ef21-4250-9a01-c468e9cd36dd`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:56,225::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:56,227::task::595::TaskManager.Task::(_updateState) Task=`c58349f1-1039-49be-b550-ffac9e4ee192`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:56,228::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:56,228::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:56,228::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:56,230::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:56,230::task::1185::TaskManager.Task::(prepare) Task=`c58349f1-1039-49be-b550-ffac9e4ee192`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:56,230::task::595::TaskManager.Task::(_updateState) Task=`c58349f1-1039-49be-b550-ffac9e4ee192`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:56,231::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:56,231::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:56,231::task::990::TaskManager.Task::(_decref) Task=`c58349f1-1039-49be-b550-ffac9e4ee192`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:57,236::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:57,237::task::595::TaskManager.Task::(_updateState) Task=`c9b5379d-28ea-4cbd-95b5-0f0c7b432889`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:57,237::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:57,238::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:57,238::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:57,238::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:57,239::task::1185::TaskManager.Task::(prepare) Task=`c9b5379d-28ea-4cbd-95b5-0f0c7b432889`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:57,239::task::595::TaskManager.Task::(_updateState) Task=`c9b5379d-28ea-4cbd-95b5-0f0c7b432889`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:57,239::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:57,239::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:57,240::task::990::TaskManager.Task::(_decref) Task=`c9b5379d-28ea-4cbd-95b5-0f0c7b432889`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:58,244::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:58,245::task::595::TaskManager.Task::(_updateState) Task=`50f6289c-5eba-4b2e-a1cc-1fd43b58479c`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:58,246::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:58,246::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:58,246::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:58,247::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:58,247::task::1185::TaskManager.Task::(prepare) Task=`50f6289c-5eba-4b2e-a1cc-1fd43b58479c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:58,247::task::595::TaskManager.Task::(_updateState) Task=`50f6289c-5eba-4b2e-a1cc-1fd43b58479c`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:58,248::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:58,248::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:58,249::task::990::TaskManager.Task::(_decref) Task=`50f6289c-5eba-4b2e-a1cc-1fd43b58479c`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:18:59,253::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:18:59,254::task::595::TaskManager.Task::(_updateState) Task=`b436693c-4ee6-4646-be23-ca19c46fab61`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:18:59,254::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:18:59,254::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:18:59,255::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:18:59,255::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:59,255::task::1185::TaskManager.Task::(prepare) Task=`b436693c-4ee6-4646-be23-ca19c46fab61`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:18:59,256::task::595::TaskManager.Task::(_updateState) Task=`b436693c-4ee6-4646-be23-ca19c46fab61`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:18:59,256::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:18:59,256::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:18:59,257::task::990::TaskManager.Task::(_decref) Task=`b436693c-4ee6-4646-be23-ca19c46fab61`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:00,267::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:00,268::task::595::TaskManager.Task::(_updateState) Task=`ca7208d2-4c79-4a81-b32b-8c09d5ec54e9`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:00,270::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:00,272::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:00,273::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:00,274::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:00,276::task::1185::TaskManager.Task::(prepare) Task=`ca7208d2-4c79-4a81-b32b-8c09d5ec54e9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:00,277::task::595::TaskManager.Task::(_updateState) Task=`ca7208d2-4c79-4a81-b32b-8c09d5ec54e9`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:00,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:00,279::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:00,279::task::990::TaskManager.Task::(_decref) Task=`ca7208d2-4c79-4a81-b32b-8c09d5ec54e9`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:01,285::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:01,287::task::595::TaskManager.Task::(_updateState) Task=`93e1561b-0f76-44d7-8888-9b43a41f50cf`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:01,287::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:01,287::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:01,288::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:01,289::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:01,289::task::1185::TaskManager.Task::(prepare) Task=`93e1561b-0f76-44d7-8888-9b43a41f50cf`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:01,290::task::595::TaskManager.Task::(_updateState) Task=`93e1561b-0f76-44d7-8888-9b43a41f50cf`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:01,291::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:01,291::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:01,292::task::990::TaskManager.Task::(_decref) Task=`93e1561b-0f76-44d7-8888-9b43a41f50cf`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:02,297::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:02,299::task::595::TaskManager.Task::(_updateState) Task=`a27312bc-86ca-4f02-9d00-f927af2bd15a`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:02,299::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:02,300::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:02,301::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:02,301::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:02,301::task::1185::TaskManager.Task::(prepare) Task=`a27312bc-86ca-4f02-9d00-f927af2bd15a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:02,302::task::595::TaskManager.Task::(_updateState) Task=`a27312bc-86ca-4f02-9d00-f927af2bd15a`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:02,304::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:02,304::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:02,305::task::990::TaskManager.Task::(_decref) Task=`a27312bc-86ca-4f02-9d00-f927af2bd15a`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:03,311::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:03,324::task::595::TaskManager.Task::(_updateState) Task=`9439906d-c076-4ae5-a489-dcf71502341d`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:03,324::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:03,324::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:03,325::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:03,325::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:03,325::task::1185::TaskManager.Task::(prepare) Task=`9439906d-c076-4ae5-a489-dcf71502341d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:03,326::task::595::TaskManager.Task::(_updateState) Task=`9439906d-c076-4ae5-a489-dcf71502341d`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:03,326::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:03,327::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:03,327::task::990::TaskManager.Task::(_decref) Task=`9439906d-c076-4ae5-a489-dcf71502341d`::ref 0 aborting False
>Thread-73::DEBUG::2014-04-11 22:19:04,054::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-83::DEBUG::2014-04-11 22:19:04,199::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 21619c8e-99ea-4813-be02-d708971e5393
>Thread-28::DEBUG::2014-04-11 22:19:04,350::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:04,352::task::595::TaskManager.Task::(_updateState) Task=`621eff95-3b6f-48d8-9087-b1267d4a46e0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:04,352::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:04,353::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:04,353::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:04,354::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:04,354::task::1185::TaskManager.Task::(prepare) Task=`621eff95-3b6f-48d8-9087-b1267d4a46e0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:04,355::task::595::TaskManager.Task::(_updateState) Task=`621eff95-3b6f-48d8-9087-b1267d4a46e0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:04,355::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:04,356::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:04,357::task::990::TaskManager.Task::(_decref) Task=`621eff95-3b6f-48d8-9087-b1267d4a46e0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:05,368::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:05,373::task::595::TaskManager.Task::(_updateState) Task=`ecb3b498-b718-4602-a697-d49be660d3f0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:05,376::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:05,377::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:05,377::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:05,377::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:05,378::task::1185::TaskManager.Task::(prepare) Task=`ecb3b498-b718-4602-a697-d49be660d3f0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:05,378::task::595::TaskManager.Task::(_updateState) Task=`ecb3b498-b718-4602-a697-d49be660d3f0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:05,381::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:05,382::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:05,384::task::990::TaskManager.Task::(_decref) Task=`ecb3b498-b718-4602-a697-d49be660d3f0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:06,389::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:06,390::task::595::TaskManager.Task::(_updateState) Task=`bea64871-97a9-4afd-9def-05d8f7416746`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:06,391::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:06,391::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:06,392::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:06,392::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:06,393::task::1185::TaskManager.Task::(prepare) Task=`bea64871-97a9-4afd-9def-05d8f7416746`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:06,393::task::595::TaskManager.Task::(_updateState) Task=`bea64871-97a9-4afd-9def-05d8f7416746`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:06,393::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:06,394::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:06,394::task::990::TaskManager.Task::(_decref) Task=`bea64871-97a9-4afd-9def-05d8f7416746`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:07,399::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:07,400::task::595::TaskManager.Task::(_updateState) Task=`5f085ca1-6348-4cb6-a429-7068ad594188`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:07,400::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:07,400::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:07,401::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:07,401::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:07,402::task::1185::TaskManager.Task::(prepare) Task=`5f085ca1-6348-4cb6-a429-7068ad594188`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:07,402::task::595::TaskManager.Task::(_updateState) Task=`5f085ca1-6348-4cb6-a429-7068ad594188`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:07,402::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:07,403::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:07,404::task::990::TaskManager.Task::(_decref) Task=`5f085ca1-6348-4cb6-a429-7068ad594188`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:08,408::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:08,421::task::595::TaskManager.Task::(_updateState) Task=`6236e55b-343a-4248-96d6-8e9bbf8fa376`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:08,422::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:08,422::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:08,423::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:08,423::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:08,424::task::1185::TaskManager.Task::(prepare) Task=`6236e55b-343a-4248-96d6-8e9bbf8fa376`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:08,424::task::595::TaskManager.Task::(_updateState) Task=`6236e55b-343a-4248-96d6-8e9bbf8fa376`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:08,425::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:08,425::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:08,426::task::990::TaskManager.Task::(_decref) Task=`6236e55b-343a-4248-96d6-8e9bbf8fa376`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:09,431::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:09,433::task::595::TaskManager.Task::(_updateState) Task=`c76847a8-f92a-4ffa-8794-7e17098aff0c`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:09,433::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:09,434::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:09,435::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:09,436::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:09,436::task::1185::TaskManager.Task::(prepare) Task=`c76847a8-f92a-4ffa-8794-7e17098aff0c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:09,437::task::595::TaskManager.Task::(_updateState) Task=`c76847a8-f92a-4ffa-8794-7e17098aff0c`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:09,437::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:09,438::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:09,439::task::990::TaskManager.Task::(_decref) Task=`c76847a8-f92a-4ffa-8794-7e17098aff0c`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:10,489::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:10,540::task::595::TaskManager.Task::(_updateState) Task=`d642f430-ed1e-4753-b2aa-a9094dc32dec`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:10,582::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:10,613::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:10,644::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:10,676::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:10,708::task::1185::TaskManager.Task::(prepare) Task=`d642f430-ed1e-4753-b2aa-a9094dc32dec`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:10,740::task::595::TaskManager.Task::(_updateState) Task=`d642f430-ed1e-4753-b2aa-a9094dc32dec`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:10,773::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:10,804::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:10,842::task::990::TaskManager.Task::(_decref) Task=`d642f430-ed1e-4753-b2aa-a9094dc32dec`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:11,956::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:11,964::task::595::TaskManager.Task::(_updateState) Task=`dea83ab0-7414-4204-a645-381143427314`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:11,971::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:11,978::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:11,985::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:11,995::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:12,001::task::1185::TaskManager.Task::(prepare) Task=`dea83ab0-7414-4204-a645-381143427314`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:12,004::task::595::TaskManager.Task::(_updateState) Task=`dea83ab0-7414-4204-a645-381143427314`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:12,012::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:12,028::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:12,037::task::990::TaskManager.Task::(_decref) Task=`dea83ab0-7414-4204-a645-381143427314`::ref 0 aborting False
>Thread-284::WARNING::2014-04-11 22:19:12,188::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-285::WARNING::2014-04-11 22:19:12,197::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-284::DEBUG::2014-04-11 22:19:12,264::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-286::WARNING::2014-04-11 22:19:12,252::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-287::WARNING::2014-04-11 22:19:12,267::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-286::DEBUG::2014-04-11 22:19:12,326::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-288::WARNING::2014-04-11 22:19:12,270::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-287::DEBUG::2014-04-11 22:19:12,345::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-289::WARNING::2014-04-11 22:19:12,276::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-288::DEBUG::2014-04-11 22:19:12,369::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-290::WARNING::2014-04-11 22:19:12,282::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-291::WARNING::2014-04-11 22:19:12,285::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-290::DEBUG::2014-04-11 22:19:12,414::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-295::WARNING::2014-04-11 22:19:12,391::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-291::DEBUG::2014-04-11 22:19:12,432::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-72::ERROR::2014-04-11 22:19:12,443::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-296::WARNING::2014-04-11 22:19:12,421::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-295::DEBUG::2014-04-11 22:19:12,452::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-297::WARNING::2014-04-11 22:19:12,427::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-81::ERROR::2014-04-11 22:19:12,461::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-72::ERROR::2014-04-11 22:19:12,469::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::ERROR::2014-04-11 22:19:12,471::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-298::WARNING::2014-04-11 22:19:12,431::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-296::DEBUG::2014-04-11 22:19:12,478::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-297::DEBUG::2014-04-11 22:19:12,489::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-81::ERROR::2014-04-11 22:19:12,496::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-301::WARNING::2014-04-11 22:19:12,499::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::ERROR::2014-04-11 22:19:12,512::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-298::DEBUG::2014-04-11 22:19:12,521::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-302::WARNING::2014-04-11 22:19:12,531::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-301::DEBUG::2014-04-11 22:19:12,545::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-303::WARNING::2014-04-11 22:19:12,532::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:19:12,552::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-302::DEBUG::2014-04-11 22:19:12,556::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-74::ERROR::2014-04-11 22:19:12,556::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-304::WARNING::2014-04-11 22:19:12,537::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles
>    sd.DOMAIN_META_DATA))
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
>    *args, **kwargs)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
>    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
>    raise Timeout()
>Timeout
>Thread-303::DEBUG::2014-04-11 22:19:12,566::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-74::ERROR::2014-04-11 22:19:12,574::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-304::DEBUG::2014-04-11 22:19:12,578::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
>Thread-82::ERROR::2014-04-11 22:19:12,587::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:19:12,590::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None)
>Thread-82::ERROR::2014-04-11 22:19:12,590::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:19:12,917::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::WARNING::2014-04-11 22:19:12,922::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8']
>9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:19:12,922::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:19:13,067::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:13,070::task::595::TaskManager.Task::(_updateState) Task=`1b4d43fc-5e71-440e-b87d-5a018cfe1219`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:13,071::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:13,071::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:13,072::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:13,073::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:13,074::task::1185::TaskManager.Task::(prepare) Task=`1b4d43fc-5e71-440e-b87d-5a018cfe1219`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:13,074::task::595::TaskManager.Task::(_updateState) Task=`1b4d43fc-5e71-440e-b87d-5a018cfe1219`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:13,075::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:13,075::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:13,075::task::990::TaskManager.Task::(_decref) Task=`1b4d43fc-5e71-440e-b87d-5a018cfe1219`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:14,080::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:14,082::task::595::TaskManager.Task::(_updateState) Task=`473667e2-7567-4d58-bbbb-b687206888c9`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:14,082::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:14,083::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:14,083::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:14,083::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:14,084::task::1185::TaskManager.Task::(prepare) Task=`473667e2-7567-4d58-bbbb-b687206888c9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:14,084::task::595::TaskManager.Task::(_updateState) Task=`473667e2-7567-4d58-bbbb-b687206888c9`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:14,084::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:14,085::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:14,085::task::990::TaskManager.Task::(_decref) Task=`473667e2-7567-4d58-bbbb-b687206888c9`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:15,090::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:15,092::task::595::TaskManager.Task::(_updateState) Task=`2ea8fd68-6e93-4ef3-83f6-063d4bcddfc0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:15,093::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:15,093::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:15,094::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:15,095::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:15,095::task::1185::TaskManager.Task::(prepare) Task=`2ea8fd68-6e93-4ef3-83f6-063d4bcddfc0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:15,095::task::595::TaskManager.Task::(_updateState) Task=`2ea8fd68-6e93-4ef3-83f6-063d4bcddfc0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:15,096::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:15,096::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:15,096::task::990::TaskManager.Task::(_decref) Task=`2ea8fd68-6e93-4ef3-83f6-063d4bcddfc0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:16,101::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:16,102::task::595::TaskManager.Task::(_updateState) Task=`1811c807-83b8-4971-8d09-5d822bec2d16`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:16,102::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:16,102::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:16,103::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:16,103::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:16,103::task::1185::TaskManager.Task::(prepare) Task=`1811c807-83b8-4971-8d09-5d822bec2d16`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:16,104::task::595::TaskManager.Task::(_updateState) Task=`1811c807-83b8-4971-8d09-5d822bec2d16`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:16,104::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:16,104::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:16,105::task::990::TaskManager.Task::(_decref) Task=`1811c807-83b8-4971-8d09-5d822bec2d16`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:17,110::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:17,112::task::595::TaskManager.Task::(_updateState) Task=`799e7cfd-5f02-4d76-a1e4-75b9e84dbbb2`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:17,113::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:17,113::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:17,113::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:17,114::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:17,114::task::1185::TaskManager.Task::(prepare) Task=`799e7cfd-5f02-4d76-a1e4-75b9e84dbbb2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:17,114::task::595::TaskManager.Task::(_updateState) Task=`799e7cfd-5f02-4d76-a1e4-75b9e84dbbb2`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:17,115::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:17,115::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:17,116::task::990::TaskManager.Task::(_decref) Task=`799e7cfd-5f02-4d76-a1e4-75b9e84dbbb2`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:18,120::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:18,121::task::595::TaskManager.Task::(_updateState) Task=`caf19566-078e-4b56-875b-957915ebe743`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:18,121::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:18,122::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:18,122::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:18,122::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:18,122::task::1185::TaskManager.Task::(prepare) Task=`caf19566-078e-4b56-875b-957915ebe743`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:18,123::task::595::TaskManager.Task::(_updateState) Task=`caf19566-078e-4b56-875b-957915ebe743`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:18,123::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:18,123::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:18,124::task::990::TaskManager.Task::(_decref) Task=`caf19566-078e-4b56-875b-957915ebe743`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:19,128::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:19,144::task::595::TaskManager.Task::(_updateState) Task=`f999d5db-b0b4-4db2-8c5d-4d31d2dd3b71`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:19,145::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:19,145::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:19,145::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:19,146::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:19,146::task::1185::TaskManager.Task::(prepare) Task=`f999d5db-b0b4-4db2-8c5d-4d31d2dd3b71`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:19,146::task::595::TaskManager.Task::(_updateState) Task=`f999d5db-b0b4-4db2-8c5d-4d31d2dd3b71`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:19,147::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:19,147::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:19,147::task::990::TaskManager.Task::(_decref) Task=`f999d5db-b0b4-4db2-8c5d-4d31d2dd3b71`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:20,152::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:20,153::task::595::TaskManager.Task::(_updateState) Task=`46e73d63-e7b8-4c7d-ad0d-876fe3ddf4fd`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:20,154::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:20,154::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:20,154::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:20,155::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:20,155::task::1185::TaskManager.Task::(prepare) Task=`46e73d63-e7b8-4c7d-ad0d-876fe3ddf4fd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:20,155::task::595::TaskManager.Task::(_updateState) Task=`46e73d63-e7b8-4c7d-ad0d-876fe3ddf4fd`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:20,156::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:20,156::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:20,156::task::990::TaskManager.Task::(_decref) Task=`46e73d63-e7b8-4c7d-ad0d-876fe3ddf4fd`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:21,161::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:21,161::task::595::TaskManager.Task::(_updateState) Task=`9bcfe900-db3f-498d-b390-459d24f37bfb`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:21,162::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:21,162::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:21,162::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:21,163::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:21,163::task::1185::TaskManager.Task::(prepare) Task=`9bcfe900-db3f-498d-b390-459d24f37bfb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:21,163::task::595::TaskManager.Task::(_updateState) Task=`9bcfe900-db3f-498d-b390-459d24f37bfb`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:21,164::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:21,164::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:21,164::task::990::TaskManager.Task::(_decref) Task=`9bcfe900-db3f-498d-b390-459d24f37bfb`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:22,168::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:22,169::task::595::TaskManager.Task::(_updateState) Task=`9ce527db-2332-443e-ad78-0aae11470a93`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:22,169::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:22,169::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:22,170::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:22,170::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:22,171::task::1185::TaskManager.Task::(prepare) Task=`9ce527db-2332-443e-ad78-0aae11470a93`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:22,171::task::595::TaskManager.Task::(_updateState) Task=`9ce527db-2332-443e-ad78-0aae11470a93`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:22,171::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:22,172::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:22,172::task::990::TaskManager.Task::(_decref) Task=`9ce527db-2332-443e-ad78-0aae11470a93`::ref 0 aborting False
>Thread-72::DEBUG::2014-04-11 22:19:22,511::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-81::DEBUG::2014-04-11 22:19:22,540::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-81::ERROR::2014-04-11 22:19:22,540::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-81::ERROR::2014-04-11 22:19:22,541::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-81::DEBUG::2014-04-11 22:19:22,541::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-81::DEBUG::2014-04-11 22:19:22,542::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-74::DEBUG::2014-04-11 22:19:22,585::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-74::ERROR::2014-04-11 22:19:22,585::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-74::ERROR::2014-04-11 22:19:22,586::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-74::DEBUG::2014-04-11 22:19:22,586::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-74::DEBUG::2014-04-11 22:19:22,587::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None)
>Thread-82::DEBUG::2014-04-11 22:19:22,609::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-82::ERROR::2014-04-11 22:19:22,610::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-82::ERROR::2014-04-11 22:19:22,611::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-82::DEBUG::2014-04-11 22:19:22,611::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-82::DEBUG::2014-04-11 22:19:22,613::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-81::DEBUG::2014-04-11 22:19:22,853::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-81::WARNING::2014-04-11 22:19:22,856::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-74::DEBUG::2014-04-11 22:19:22,904::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5
>Thread-82::DEBUG::2014-04-11 22:19:22,983::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-74::WARNING::2014-04-11 22:19:22,988::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce']
>Thread-82::WARNING::2014-04-11 22:19:22,999::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-82::DEBUG::2014-04-11 22:19:23,005::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-28::DEBUG::2014-04-11 22:19:23,256::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:23,264::task::595::TaskManager.Task::(_updateState) Task=`15267231-eca5-4bf4-87e4-ad6040bd6b79`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:23,264::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:23,264::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:23,265::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:23,265::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:23,266::task::1185::TaskManager.Task::(prepare) Task=`15267231-eca5-4bf4-87e4-ad6040bd6b79`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:23,266::task::595::TaskManager.Task::(_updateState) Task=`15267231-eca5-4bf4-87e4-ad6040bd6b79`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:23,266::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:23,267::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:23,268::task::990::TaskManager.Task::(_decref) Task=`15267231-eca5-4bf4-87e4-ad6040bd6b79`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:24,272::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:24,273::task::595::TaskManager.Task::(_updateState) Task=`7177d31b-3931-48b6-8262-f4a6e81a492b`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:24,274::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:24,274::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:24,274::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:24,275::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:24,275::task::1185::TaskManager.Task::(prepare) Task=`7177d31b-3931-48b6-8262-f4a6e81a492b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:24,275::task::595::TaskManager.Task::(_updateState) Task=`7177d31b-3931-48b6-8262-f4a6e81a492b`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:24,276::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:24,276::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:24,276::task::990::TaskManager.Task::(_decref) Task=`7177d31b-3931-48b6-8262-f4a6e81a492b`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:25,282::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:25,284::task::595::TaskManager.Task::(_updateState) Task=`e6ca6b07-1d05-464a-adb8-fa158b6f0ad0`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:25,284::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:25,285::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:25,285::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:25,286::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:25,286::task::1185::TaskManager.Task::(prepare) Task=`e6ca6b07-1d05-464a-adb8-fa158b6f0ad0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:25,286::task::595::TaskManager.Task::(_updateState) Task=`e6ca6b07-1d05-464a-adb8-fa158b6f0ad0`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:25,287::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:25,287::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:25,287::task::990::TaskManager.Task::(_decref) Task=`e6ca6b07-1d05-464a-adb8-fa158b6f0ad0`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:26,293::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:26,293::task::595::TaskManager.Task::(_updateState) Task=`add95d90-907c-40e9-a455-b54cc69ca8a7`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:26,295::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:26,295::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:26,296::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:26,297::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:26,297::task::1185::TaskManager.Task::(prepare) Task=`add95d90-907c-40e9-a455-b54cc69ca8a7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:26,298::task::595::TaskManager.Task::(_updateState) Task=`add95d90-907c-40e9-a455-b54cc69ca8a7`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:26,298::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:26,298::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:26,299::task::990::TaskManager.Task::(_decref) Task=`add95d90-907c-40e9-a455-b54cc69ca8a7`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:27,303::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:27,304::task::595::TaskManager.Task::(_updateState) Task=`b6814913-1c85-46f5-b2a5-fc08e01bf832`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:27,305::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:27,305::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:27,305::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:27,306::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:27,306::task::1185::TaskManager.Task::(prepare) Task=`b6814913-1c85-46f5-b2a5-fc08e01bf832`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:27,307::task::595::TaskManager.Task::(_updateState) Task=`b6814913-1c85-46f5-b2a5-fc08e01bf832`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:27,307::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:27,307::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:27,308::task::990::TaskManager.Task::(_decref) Task=`b6814913-1c85-46f5-b2a5-fc08e01bf832`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:28,312::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:28,313::task::595::TaskManager.Task::(_updateState) Task=`aba5a048-3300-4ab3-95ed-a02805f09d5a`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:28,313::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:28,313::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:28,314::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:28,314::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:28,314::task::1185::TaskManager.Task::(prepare) Task=`aba5a048-3300-4ab3-95ed-a02805f09d5a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:28,314::task::595::TaskManager.Task::(_updateState) Task=`aba5a048-3300-4ab3-95ed-a02805f09d5a`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:28,315::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:28,315::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:28,315::task::990::TaskManager.Task::(_decref) Task=`aba5a048-3300-4ab3-95ed-a02805f09d5a`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:29,319::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:29,320::task::595::TaskManager.Task::(_updateState) Task=`6c9b0c83-377e-482f-bd14-546ee9f72c39`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:29,321::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:29,321::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:29,321::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:29,321::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:29,322::task::1185::TaskManager.Task::(prepare) Task=`6c9b0c83-377e-482f-bd14-546ee9f72c39`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:29,322::task::595::TaskManager.Task::(_updateState) Task=`6c9b0c83-377e-482f-bd14-546ee9f72c39`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:29,322::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:29,322::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:29,323::task::990::TaskManager.Task::(_decref) Task=`6c9b0c83-377e-482f-bd14-546ee9f72c39`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:30,328::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:30,329::task::595::TaskManager.Task::(_updateState) Task=`0d87e83f-9cee-446c-bb09-f1a030e2f40d`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:30,330::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:30,330::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:30,331::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:30,332::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:30,332::task::1185::TaskManager.Task::(prepare) Task=`0d87e83f-9cee-446c-bb09-f1a030e2f40d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:30,332::task::595::TaskManager.Task::(_updateState) Task=`0d87e83f-9cee-446c-bb09-f1a030e2f40d`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:30,333::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:30,334::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:30,335::task::990::TaskManager.Task::(_decref) Task=`0d87e83f-9cee-446c-bb09-f1a030e2f40d`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:31,341::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:31,342::task::595::TaskManager.Task::(_updateState) Task=`7261bcd7-3354-4dbe-8386-56c49a13461a`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:31,343::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:31,343::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:31,343::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:31,344::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:31,345::task::1185::TaskManager.Task::(prepare) Task=`7261bcd7-3354-4dbe-8386-56c49a13461a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:31,345::task::595::TaskManager.Task::(_updateState) Task=`7261bcd7-3354-4dbe-8386-56c49a13461a`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:31,346::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:31,346::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:31,347::task::990::TaskManager.Task::(_decref) Task=`7261bcd7-3354-4dbe-8386-56c49a13461a`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:32,351::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:32,352::task::595::TaskManager.Task::(_updateState) Task=`e2eecfc3-daf6-499e-8d21-cadff077adcb`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:32,353::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:32,353::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:32,354::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:32,354::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:32,355::task::1185::TaskManager.Task::(prepare) Task=`e2eecfc3-daf6-499e-8d21-cadff077adcb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:32,355::task::595::TaskManager.Task::(_updateState) Task=`e2eecfc3-daf6-499e-8d21-cadff077adcb`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:32,356::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:32,357::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:32,357::task::990::TaskManager.Task::(_decref) Task=`e2eecfc3-daf6-499e-8d21-cadff077adcb`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:33,362::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:33,362::task::595::TaskManager.Task::(_updateState) Task=`00fb333b-4c18-455e-85aa-2578acb9e0fa`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:33,363::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:33,364::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:33,364::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:33,364::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:33,365::task::1185::TaskManager.Task::(prepare) Task=`00fb333b-4c18-455e-85aa-2578acb9e0fa`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:33,365::task::595::TaskManager.Task::(_updateState) Task=`00fb333b-4c18-455e-85aa-2578acb9e0fa`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:33,365::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:33,365::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:33,366::task::990::TaskManager.Task::(_decref) Task=`00fb333b-4c18-455e-85aa-2578acb9e0fa`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:34,370::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:34,371::task::595::TaskManager.Task::(_updateState) Task=`6b17fed8-3f4b-4368-b881-9403b047d927`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:34,371::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:34,371::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:34,372::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:34,372::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:34,372::task::1185::TaskManager.Task::(prepare) Task=`6b17fed8-3f4b-4368-b881-9403b047d927`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:34,373::task::595::TaskManager.Task::(_updateState) Task=`6b17fed8-3f4b-4368-b881-9403b047d927`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:34,373::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:34,373::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:34,373::task::990::TaskManager.Task::(_decref) Task=`6b17fed8-3f4b-4368-b881-9403b047d927`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:35,379::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:35,380::task::595::TaskManager.Task::(_updateState) Task=`3a766c95-e5c7-4ce4-9161-a2276b9eb1e6`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:35,382::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:35,382::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:35,382::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:35,383::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:35,383::task::1185::TaskManager.Task::(prepare) Task=`3a766c95-e5c7-4ce4-9161-a2276b9eb1e6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:35,385::task::595::TaskManager.Task::(_updateState) Task=`3a766c95-e5c7-4ce4-9161-a2276b9eb1e6`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:35,386::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:35,386::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:35,387::task::990::TaskManager.Task::(_decref) Task=`3a766c95-e5c7-4ce4-9161-a2276b9eb1e6`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:36,392::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:36,393::task::595::TaskManager.Task::(_updateState) Task=`504a5a5a-93a0-426d-ba3f-472111029615`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:36,393::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:36,393::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:36,394::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:36,394::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:36,394::task::1185::TaskManager.Task::(prepare) Task=`504a5a5a-93a0-426d-ba3f-472111029615`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:36,395::task::595::TaskManager.Task::(_updateState) Task=`504a5a5a-93a0-426d-ba3f-472111029615`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:36,395::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:36,395::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:36,395::task::990::TaskManager.Task::(_decref) Task=`504a5a5a-93a0-426d-ba3f-472111029615`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:37,400::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:37,401::task::595::TaskManager.Task::(_updateState) Task=`f7874fcc-5d27-465e-ade6-0404a41d1b4b`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:37,401::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:37,402::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:37,402::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:37,402::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:37,402::task::1185::TaskManager.Task::(prepare) Task=`f7874fcc-5d27-465e-ade6-0404a41d1b4b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:37,403::task::595::TaskManager.Task::(_updateState) Task=`f7874fcc-5d27-465e-ade6-0404a41d1b4b`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:37,403::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:37,403::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:37,404::task::990::TaskManager.Task::(_decref) Task=`f7874fcc-5d27-465e-ade6-0404a41d1b4b`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:38,408::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:38,408::task::595::TaskManager.Task::(_updateState) Task=`d279ba4f-bb4e-45df-ba3c-bae03ca7fa9b`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:38,409::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:38,409::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:38,409::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:38,410::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:38,410::task::1185::TaskManager.Task::(prepare) Task=`d279ba4f-bb4e-45df-ba3c-bae03ca7fa9b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:38,410::task::595::TaskManager.Task::(_updateState) Task=`d279ba4f-bb4e-45df-ba3c-bae03ca7fa9b`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:38,411::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:38,411::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:38,411::task::990::TaskManager.Task::(_decref) Task=`d279ba4f-bb4e-45df-ba3c-bae03ca7fa9b`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:39,415::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:39,416::task::595::TaskManager.Task::(_updateState) Task=`ee7f9449-acec-49c8-9d4c-b0c751e23c84`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:39,417::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:39,417::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:39,417::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:39,418::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:39,418::task::1185::TaskManager.Task::(prepare) Task=`ee7f9449-acec-49c8-9d4c-b0c751e23c84`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:39,418::task::595::TaskManager.Task::(_updateState) Task=`ee7f9449-acec-49c8-9d4c-b0c751e23c84`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:39,418::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:39,419::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:39,419::task::990::TaskManager.Task::(_decref) Task=`ee7f9449-acec-49c8-9d4c-b0c751e23c84`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:40,433::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:40,434::task::595::TaskManager.Task::(_updateState) Task=`e9859e0e-53e3-440f-9aca-3da5152b3255`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:40,435::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:40,435::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:40,435::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:40,436::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:40,436::task::1185::TaskManager.Task::(prepare) Task=`e9859e0e-53e3-440f-9aca-3da5152b3255`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:40,437::task::595::TaskManager.Task::(_updateState) Task=`e9859e0e-53e3-440f-9aca-3da5152b3255`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:40,437::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:40,438::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:40,438::task::990::TaskManager.Task::(_decref) Task=`e9859e0e-53e3-440f-9aca-3da5152b3255`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:41,444::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:41,445::task::595::TaskManager.Task::(_updateState) Task=`39f37e0a-7d01-4950-8992-8701847b78ef`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:41,445::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:41,445::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:41,447::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:41,448::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:41,448::task::1185::TaskManager.Task::(prepare) Task=`39f37e0a-7d01-4950-8992-8701847b78ef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:41,448::task::595::TaskManager.Task::(_updateState) Task=`39f37e0a-7d01-4950-8992-8701847b78ef`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:41,449::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:41,449::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:41,450::task::990::TaskManager.Task::(_decref) Task=`39f37e0a-7d01-4950-8992-8701847b78ef`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:42,454::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e]
>Thread-28::DEBUG::2014-04-11 22:19:42,455::task::595::TaskManager.Task::(_updateState) Task=`e3f1f0fd-5db8-4cb4-804a-f6a77897b340`::moving from state init -> state preparing
>Thread-28::INFO::2014-04-11 22:19:42,456::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None)
>Thread-28::DEBUG::2014-04-11 22:19:42,456::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5
>Thread-28::DEBUG::2014-04-11 22:19:42,456::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}
>Thread-28::INFO::2014-04-11 22:19:42,457::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:42,457::task::1185::TaskManager.Task::(prepare) Task=`e3f1f0fd-5db8-4cb4-804a-f6a77897b340`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}}
>Thread-28::DEBUG::2014-04-11 22:19:42,457::task::595::TaskManager.Task::(_updateState) Task=`e3f1f0fd-5db8-4cb4-804a-f6a77897b340`::moving from state preparing -> state finished
>Thread-28::DEBUG::2014-04-11 22:19:42,458::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-28::DEBUG::2014-04-11 22:19:42,458::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-28::DEBUG::2014-04-11 22:19:42,458::task::990::TaskManager.Task::(_decref) Task=`e3f1f0fd-5db8-4cb4-804a-f6a77897b340`::ref 0 aborting False
>Thread-28::DEBUG::2014-04-11 22:19:43,463::BindingXMLRPC::251::vds::(wrapper) clientMainThread::INFO::2014-04-11 22:22:43,596::vdsm::119::vds::(run) (PID: 4571) I am the actual vdsm 4.14.6-12.git49cacf0.el6 ashtivh01.suprtekstic.com (2.6.32-431.11.2.el6.x86_64)
>MainThread::DEBUG::2014-04-11 22:22:43,605::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage'
>MainThread::DEBUG::2014-04-11 22:22:43,606::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
>MainThread::WARNING::2014-04-11 22:22:43,612::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists
>MainThread::DEBUG::2014-04-11 22:22:45,249::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
>MainThread::DEBUG::2014-04-11 22:22:45,286::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving
>MainThread::DEBUG::2014-04-11 22:22:45,286::hsm::424::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
>MainThread::DEBUG::2014-04-11 22:22:45,554::hsm::424::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
>MainThread::DEBUG::2014-04-11 22:22:45,555::hsm::448::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
>MainThread::DEBUG::2014-04-11 22:22:45,561::hsm::480::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
>MainThread::DEBUG::2014-04-11 22:22:45,562::hsm::481::Storage.HSM::(__cleanStorageRepository) Mount list: []
>MainThread::DEBUG::2014-04-11 22:22:45,562::hsm::483::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
>MainThread::DEBUG::2014-04-11 22:22:45,596::hsm::526::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
>storageRefresh::DEBUG::2014-04-11 22:22:45,598::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>MainThread::INFO::2014-04-11 22:22:45,601::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
>storageRefresh::DEBUG::2014-04-11 22:22:45,601::misc::764::SamplingMethod::(__call__) Got in to sampling method
>storageRefresh::DEBUG::2014-04-11 22:22:45,803::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>storageRefresh::DEBUG::2014-04-11 22:22:45,803::misc::764::SamplingMethod::(__call__) Got in to sampling method
>storageRefresh::DEBUG::2014-04-11 22:22:45,808::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>storageRefresh::DEBUG::2014-04-11 22:22:45,809::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:22:45,892::misc::772::SamplingMethod::(__call__) Returning last result
>storageRefresh::DEBUG::2014-04-11 22:22:45,893::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>MainThread::DEBUG::2014-04-11 22:22:45,968::task::595::TaskManager.Task::(_updateState) Task=`4b7ade79-a63d-4da2-b641-956821a6d085`::moving from state init -> state preparing
>MainThread::INFO::2014-04-11 22:22:45,968::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x2b26290>>)
>MainThread::INFO::2014-04-11 22:22:45,969::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
>MainThread::DEBUG::2014-04-11 22:22:45,969::task::1185::TaskManager.Task::(prepare) Task=`4b7ade79-a63d-4da2-b641-956821a6d085`::finished: None
>MainThread::DEBUG::2014-04-11 22:22:45,969::task::595::TaskManager.Task::(_updateState) Task=`4b7ade79-a63d-4da2-b641-956821a6d085`::moving from state preparing -> state finished
>MainThread::DEBUG::2014-04-11 22:22:45,969::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>MainThread::DEBUG::2014-04-11 22:22:45,970::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>MainThread::DEBUG::2014-04-11 22:22:45,970::task::990::TaskManager.Task::(_decref) Task=`4b7ade79-a63d-4da2-b641-956821a6d085`::ref 0 aborting False
>MainThread::INFO::2014-04-11 22:22:45,986::momIF::47::MOM::(__init__) Starting up MOM
>MainThread::INFO::2014-04-11 22:22:46,004::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds.
>VM Channels Listener::INFO::2014-04-11 22:22:46,006::vmChannels::170::vds::(run) Starting VM channels listener thread.
>clientIFinit::DEBUG::2014-04-11 22:22:46,052::libvirtconnection::145::root::(get) trying to connect libvirt
>storageRefresh::DEBUG::2014-04-11 22:22:46,055::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:22:46,056::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,057::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,063::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,064::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,064::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,064::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,065::misc::772::SamplingMethod::(__call__) Returning last result
>storageRefresh::DEBUG::2014-04-11 22:22:46,065::lvm::324::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,070::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
>MainThread::WARNING::2014-04-11 22:22:46,081::clientIF::181::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
>storageRefresh::DEBUG::2014-04-11 22:22:46,410::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:22:46,415::lvm::349::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,419::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,429::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:22:46,718::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:22:46,718::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,719::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:22:46,938::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:22:46,939::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>storageRefresh::DEBUG::2014-04-11 22:22:46,941::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
>storageRefresh::DEBUG::2014-04-11 22:22:47,182::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
>storageRefresh::DEBUG::2014-04-11 22:22:47,183::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>storageRefresh::WARNING::2014-04-11 22:22:47,184::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists
>storageRefresh::DEBUG::2014-04-11 22:22:47,185::hsm::392::Storage.HSM::(storageRefresh) HSM is ready
>Thread-13::DEBUG::2014-04-11 22:22:50,198::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [744e80e3]
>Thread-13::DEBUG::2014-04-11 22:22:50,199::task::595::TaskManager.Task::(_updateState) Task=`114efa06-7908-4145-89a4-9139d71e4075`::moving from state init -> state preparing
>Thread-13::INFO::2014-04-11 22:22:50,199::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-13::ERROR::2014-04-11 22:22:50,200::task::866::TaskManager.Task::(_setError) Task=`114efa06-7908-4145-89a4-9139d71e4075`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
>    raise se.SpmStatusError()
>SpmStatusError: Not SPM: ()
>Thread-13::DEBUG::2014-04-11 22:22:50,227::task::885::TaskManager.Task::(_run) Task=`114efa06-7908-4145-89a4-9139d71e4075`::Task._run: 114efa06-7908-4145-89a4-9139d71e4075 () {} failed - stopping task
>Thread-13::DEBUG::2014-04-11 22:22:50,227::task::1211::TaskManager.Task::(stop) Task=`114efa06-7908-4145-89a4-9139d71e4075`::stopping in state preparing (force False)
>Thread-13::DEBUG::2014-04-11 22:22:50,227::task::990::TaskManager.Task::(_decref) Task=`114efa06-7908-4145-89a4-9139d71e4075`::ref 1 aborting True
>Thread-13::INFO::2014-04-11 22:22:50,228::task::1168::TaskManager.Task::(prepare) Task=`114efa06-7908-4145-89a4-9139d71e4075`::aborting: Task is aborted: 'Not SPM' - code 654
>Thread-13::DEBUG::2014-04-11 22:22:50,228::task::1173::TaskManager.Task::(prepare) Task=`114efa06-7908-4145-89a4-9139d71e4075`::Prepare: aborted: Not SPM
>Thread-13::DEBUG::2014-04-11 22:22:50,228::task::990::TaskManager.Task::(_decref) Task=`114efa06-7908-4145-89a4-9139d71e4075`::ref 0 aborting True
>Thread-13::DEBUG::2014-04-11 22:22:50,229::task::925::TaskManager.Task::(_doAbort) Task=`114efa06-7908-4145-89a4-9139d71e4075`::Task._doAbort: force False
>Thread-13::DEBUG::2014-04-11 22:22:50,229::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::DEBUG::2014-04-11 22:22:50,229::task::595::TaskManager.Task::(_updateState) Task=`114efa06-7908-4145-89a4-9139d71e4075`::moving from state preparing -> state aborting
>Thread-13::DEBUG::2014-04-11 22:22:50,229::task::550::TaskManager.Task::(__state_aborting) Task=`114efa06-7908-4145-89a4-9139d71e4075`::_aborting: recover policy none
>Thread-13::DEBUG::2014-04-11 22:22:50,230::task::595::TaskManager.Task::(_updateState) Task=`114efa06-7908-4145-89a4-9139d71e4075`::moving from state aborting -> state failed
>Thread-13::DEBUG::2014-04-11 22:22:50,230::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-13::DEBUG::2014-04-11 22:22:50,230::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-13::ERROR::2014-04-11 22:22:50,230::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
>Thread-14::DEBUG::2014-04-11 22:53:34,883::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getCapabilities with () {} flowID [1e4e67ba]
>Thread-14::DEBUG::2014-04-11 22:53:35,049::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:35,065::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:35,106::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:35,117::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:35,123::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:35,132::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:35,138::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.5' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:35,147::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:35,152::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.6' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:35,160::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:35,470::caps::458::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-14::DEBUG::2014-04-11 22:53:35,470::caps::458::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-14::DEBUG::2014-04-11 22:53:35,472::caps::458::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-14::DEBUG::2014-04-11 22:53:35,479::caps::458::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-14::DEBUG::2014-04-11 22:53:35,480::caps::458::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-14::DEBUG::2014-04-11 22:53:35,480::caps::458::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-14::DEBUG::2014-04-11 22:53:35,490::caps::458::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-14::DEBUG::2014-04-11 22:53:35,490::caps::458::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-14::DEBUG::2014-04-11 22:53:35,491::caps::458::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-14::DEBUG::2014-04-11 22:53:35,491::caps::458::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-14::DEBUG::2014-04-11 22:53:35,492::caps::458::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-14::DEBUG::2014-04-11 22:53:35,492::caps::458::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-14::DEBUG::2014-04-11 22:53:35,519::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:93188a588127'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395791995.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '12.git49cacf0.el6', 'buildtime': 1397150575L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz', 'hooks': {'after_vm_start': {'50_promisc': {'md5': '5ea469fa06dd7cd1e52508672a7519db'}}, 'before_vm_migrate_source': {'50_sriov': {'md5': '47a59103b7c8b1439ae17dcd5ed197cd'}, '50_hostusb': {'md5': 'e50a0975acd8bafd0052b65a787c2fdd'}, '50_scratchpad': {'md5': 'be90995f959c60c5110c6a0bef67c7b8'}, '50_floppy': {'md5': 'c68fb91d44105c8b6f2d80566314027a'}}, 'after_device_destroy': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_vm_destroy': {'50_promisc': {'md5': 'dae0434ea8ca2492422a6fde5cc47a1e'}}, 'before_vm_start': {'50_sriov': {'md5': '3ebc60cd2e4eb089820102285fad7c45'}, '50_vmfex': {'md5': '9f5abb892ddb6b3daa779985d38d9f55'}, '50_qos': {'md5': 'cdf6f89d277a886691812ce1ec2fe105'}, '50_scratchpad': {'md5': '178171eb62ba95fc548e2450013f5ad5'}, '50_numa': {'md5': '5008c2826714ac5b63748780aabd2f25'}, '50_floppy': {'md5': '202fe18705a7d4c50c40c126e8f8dbe8'}, '50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}, '50_pincpu': {'md5': '0b5fb99ff0e7acb9ad534b87c02c59e3'}, '50_hugepages': {'md5': '6713d703f9d316f1664145b700cd7904'}, '50_fileinject': {'md5': '290dafd45601e010e0f496fa5b889a76'}, '50_macspoof': {'md5': '25deea559772719b90117f2e1009d823'}, '60_checkimages': {'md5': 'abd32c723a90ab2816e2163f5e02faf7'}, '10_faqemu': {'md5': 'e3cb338be6b9075a04bed0e7400ab7e2'}, '50_directlun': {'md5': 'dba5d80fdb7119ee0f4a40c26eac8448'}, '50_smbios': {'md5': '1a6bd8eddc3a86943e558bc4d3e73bd6'}, '50_vmdisk': {'md5': '67617c6e2d796388ddc6c065a97f714e'}, '50_hostusb': {'md5': '21c58568f3eae4f0d091005e0eff891a'}, '50_isolatedprivatevlan': {'md5': '0a25d7cd0b807ea5e0dfc3318ee98c91'}}, 'after_device_migrate_destination': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'after_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_vm_migrate_destination': {'50_vmfex': {'md5': 'eac2633f47fb309b4a5b7fe76531baec'}, '50_hugepages': {'md5': '6ddc484746297308b9f9acc5fcabaf10'}, '50_directlun': {'md5': '8a29ce41de079630cb58d48e280d55c2'}}, 'after_nic_hotunplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_vm_destroy': {'50_sriov': {'md5': '5890b2899d450495a62f3611fad3c526'}, '50_directlun': {'md5': '6d05c19e68a141eeb7ee5981d68c0cfa'}, '50_hostusb': {'md5': '0a37bf0a26546e3ab5e424b13653ce2b'}, '50_hugepages': {'md5': '4a980b78f8ee908bf2c61ea9bf156f4a'}, '50_scratchpad': {'md5': '316d21a669f97e83b485b45ab0b23eee'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0', 'addr': '172.16.10.1', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'bridged': False, 'interface': u'bond0', 'ipv6gateway': '::', 'gateway': '172.16.0.1'}, 'Blue': {'iface': 'Blue', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.6']}, 'Infrastructure': {'iface': 'Infrastructure', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.4']}, 'migration': {'iface': u'bond0.5', 'addr': '', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'bridged': False, 'interface': u'bond0.5', 'ipv6gateway': '::', 'gateway': ''}, 'Red': {'iface': 'Red', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.3']}}, 'bridges': {'Blue': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.6']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'Infrastructure': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.4']}, 'Red': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.3']}}, 'uuid': '4C4C4544-0059-5310-8031-B9C04F445A31', 'lastClientIface': 'bond0', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:5F', 'MASTER': 'bond0', 'DEVICE': 'em1', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:5F', 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:60', 'MASTER': 'bond0', 'DEVICE': 'em2', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:60', 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '12', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,lahf_lm,ida,arat,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:93188a588127', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '255.255.0.0', 'addr': '172.16.10.1', 'slaves': ['em1', 'em2'], 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.10.1', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond0', 'MTU': '1500', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32057', 'cpuSpeed': '1900.070', 'version_name': 'Snow Man', 'vlans': {'bond0.3': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Red', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 3, 'mtu': '1500'}, 'bond0.4': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Infrastructure', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 4, 'mtu': '1500'}, 'bond0.5': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'DEVICE': 'bond0.5', 'NM_CONTROLLED': 'no', 'VLAN': 'yes', 'DEFROUTE': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 5, 'mtu': '1500'}, 'bond0.6': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Blue', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.6', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 6, 'mtu': '1500'}}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '24', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.10.1'}}
>Thread-14::DEBUG::2014-04-11 22:53:35,557::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getHardwareInfo with () {} flowID [1e4e67ba]
>Thread-14::DEBUG::2014-04-11 22:53:35,769::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R420', 'systemSerialNumber': '9YS1DZ1', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0059-5310-8031-B9C04F445A31', 'systemManufacturer': 'Dell Inc.'}}
>Thread-14::DEBUG::2014-04-11 22:53:39,189::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getCapabilities with () {}
>Thread-14::DEBUG::2014-04-11 22:53:39,194::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,205::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,241::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,252::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,258::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,268::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,274::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.5' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,283::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,288::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.6' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,297::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,319::caps::458::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-14::DEBUG::2014-04-11 22:53:39,320::caps::458::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-14::DEBUG::2014-04-11 22:53:39,321::caps::458::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-14::DEBUG::2014-04-11 22:53:39,329::caps::458::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-14::DEBUG::2014-04-11 22:53:39,329::caps::458::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-14::DEBUG::2014-04-11 22:53:39,330::caps::458::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-14::DEBUG::2014-04-11 22:53:39,340::caps::458::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-14::DEBUG::2014-04-11 22:53:39,340::caps::458::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-14::DEBUG::2014-04-11 22:53:39,341::caps::458::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-14::DEBUG::2014-04-11 22:53:39,341::caps::458::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-14::DEBUG::2014-04-11 22:53:39,342::caps::458::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-14::DEBUG::2014-04-11 22:53:39,342::caps::458::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-14::DEBUG::2014-04-11 22:53:39,349::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:93188a588127'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395791995.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '12.git49cacf0.el6', 'buildtime': 1397150575L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz', 'hooks': {'after_vm_start': {'50_promisc': {'md5': '5ea469fa06dd7cd1e52508672a7519db'}}, 'before_vm_migrate_source': {'50_sriov': {'md5': '47a59103b7c8b1439ae17dcd5ed197cd'}, '50_hostusb': {'md5': 'e50a0975acd8bafd0052b65a787c2fdd'}, '50_scratchpad': {'md5': 'be90995f959c60c5110c6a0bef67c7b8'}, '50_floppy': {'md5': 'c68fb91d44105c8b6f2d80566314027a'}}, 'after_device_destroy': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_vm_destroy': {'50_promisc': {'md5': 'dae0434ea8ca2492422a6fde5cc47a1e'}}, 'before_vm_start': {'50_sriov': {'md5': '3ebc60cd2e4eb089820102285fad7c45'}, '50_vmfex': {'md5': '9f5abb892ddb6b3daa779985d38d9f55'}, '50_qos': {'md5': 'cdf6f89d277a886691812ce1ec2fe105'}, '50_scratchpad': {'md5': '178171eb62ba95fc548e2450013f5ad5'}, '50_numa': {'md5': '5008c2826714ac5b63748780aabd2f25'}, '50_floppy': {'md5': '202fe18705a7d4c50c40c126e8f8dbe8'}, '50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}, '50_pincpu': {'md5': '0b5fb99ff0e7acb9ad534b87c02c59e3'}, '50_hugepages': {'md5': '6713d703f9d316f1664145b700cd7904'}, '50_fileinject': {'md5': '290dafd45601e010e0f496fa5b889a76'}, '50_macspoof': {'md5': '25deea559772719b90117f2e1009d823'}, '60_checkimages': {'md5': 'abd32c723a90ab2816e2163f5e02faf7'}, '10_faqemu': {'md5': 'e3cb338be6b9075a04bed0e7400ab7e2'}, '50_directlun': {'md5': 'dba5d80fdb7119ee0f4a40c26eac8448'}, '50_smbios': {'md5': '1a6bd8eddc3a86943e558bc4d3e73bd6'}, '50_vmdisk': {'md5': '67617c6e2d796388ddc6c065a97f714e'}, '50_hostusb': {'md5': '21c58568f3eae4f0d091005e0eff891a'}, '50_isolatedprivatevlan': {'md5': '0a25d7cd0b807ea5e0dfc3318ee98c91'}}, 'after_device_migrate_destination': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'after_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_vm_migrate_destination': {'50_vmfex': {'md5': 'eac2633f47fb309b4a5b7fe76531baec'}, '50_hugepages': {'md5': '6ddc484746297308b9f9acc5fcabaf10'}, '50_directlun': {'md5': '8a29ce41de079630cb58d48e280d55c2'}}, 'after_nic_hotunplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_vm_destroy': {'50_sriov': {'md5': '5890b2899d450495a62f3611fad3c526'}, '50_directlun': {'md5': '6d05c19e68a141eeb7ee5981d68c0cfa'}, '50_hostusb': {'md5': '0a37bf0a26546e3ab5e424b13653ce2b'}, '50_hugepages': {'md5': '4a980b78f8ee908bf2c61ea9bf156f4a'}, '50_scratchpad': {'md5': '316d21a669f97e83b485b45ab0b23eee'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0', 'addr': '172.16.10.1', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'bridged': False, 'interface': u'bond0', 'ipv6gateway': '::', 'gateway': '172.16.0.1'}, 'Blue': {'iface': 'Blue', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.6']}, 'Infrastructure': {'iface': 'Infrastructure', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.4']}, 'migration': {'iface': u'bond0.5', 'addr': '', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'bridged': False, 'interface': u'bond0.5', 'ipv6gateway': '::', 'gateway': ''}, 'Red': {'iface': 'Red', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.3']}}, 'bridges': {'Blue': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.6']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'Infrastructure': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.4']}, 'Red': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.3']}}, 'uuid': '4C4C4544-0059-5310-8031-B9C04F445A31', 'lastClientIface': 'bond0', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:5F', 'MASTER': 'bond0', 'DEVICE': 'em1', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:5F', 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:60', 'MASTER': 'bond0', 'DEVICE': 'em2', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:60', 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '12', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,lahf_lm,ida,arat,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:93188a588127', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '255.255.0.0', 'addr': '172.16.10.1', 'slaves': ['em1', 'em2'], 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.10.1', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond0', 'MTU': '1500', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32057', 'cpuSpeed': '1900.070', 'version_name': 'Snow Man', 'vlans': {'bond0.3': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Red', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 3, 'mtu': '1500'}, 'bond0.4': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Infrastructure', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 4, 'mtu': '1500'}, 'bond0.5': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'DEVICE': 'bond0.5', 'NM_CONTROLLED': 'no', 'VLAN': 'yes', 'DEFROUTE': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 5, 'mtu': '1500'}, 'bond0.6': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Blue', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.6', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 6, 'mtu': '1500'}}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '24', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.10.1'}}
>Thread-14::DEBUG::2014-04-11 22:53:39,381::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getHardwareInfo with () {}
>Thread-14::DEBUG::2014-04-11 22:53:39,383::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R420', 'systemSerialNumber': '9YS1DZ1', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0059-5310-8031-B9C04F445A31', 'systemManufacturer': 'Dell Inc.'}}
>Thread-14::DEBUG::2014-04-11 22:53:39,462::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getCapabilities with () {}
>Thread-14::DEBUG::2014-04-11 22:53:39,470::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,482::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,521::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,530::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,537::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,546::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,553::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.5' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,562::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,566::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.6' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:53:39,576::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:53:39,598::caps::458::root::(_getKeyPackages) rpm package glusterfs-rdma not found
>Thread-14::DEBUG::2014-04-11 22:53:39,599::caps::458::root::(_getKeyPackages) rpm package glusterfs-fuse not found
>Thread-14::DEBUG::2014-04-11 22:53:39,601::caps::458::root::(_getKeyPackages) rpm package gluster-swift not found
>Thread-14::DEBUG::2014-04-11 22:53:39,608::caps::458::root::(_getKeyPackages) rpm package gluster-swift-object not found
>Thread-14::DEBUG::2014-04-11 22:53:39,609::caps::458::root::(_getKeyPackages) rpm package glusterfs not found
>Thread-14::DEBUG::2014-04-11 22:53:39,609::caps::458::root::(_getKeyPackages) rpm package gluster-swift-plugin not found
>Thread-14::DEBUG::2014-04-11 22:53:39,619::caps::458::root::(_getKeyPackages) rpm package gluster-swift-account not found
>Thread-14::DEBUG::2014-04-11 22:53:39,620::caps::458::root::(_getKeyPackages) rpm package gluster-swift-proxy not found
>Thread-14::DEBUG::2014-04-11 22:53:39,620::caps::458::root::(_getKeyPackages) rpm package gluster-swift-doc not found
>Thread-14::DEBUG::2014-04-11 22:53:39,621::caps::458::root::(_getKeyPackages) rpm package glusterfs-server not found
>Thread-14::DEBUG::2014-04-11 22:53:39,621::caps::458::root::(_getKeyPackages) rpm package gluster-swift-container not found
>Thread-14::DEBUG::2014-04-11 22:53:39,622::caps::458::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found
>Thread-14::DEBUG::2014-04-11 22:53:39,630::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:93188a588127'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395791995.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '12.git49cacf0.el6', 'buildtime': 1397150575L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz', 'hooks': {'after_vm_start': {'50_promisc': {'md5': '5ea469fa06dd7cd1e52508672a7519db'}}, 'before_vm_migrate_source': {'50_sriov': {'md5': '47a59103b7c8b1439ae17dcd5ed197cd'}, '50_hostusb': {'md5': 'e50a0975acd8bafd0052b65a787c2fdd'}, '50_scratchpad': {'md5': 'be90995f959c60c5110c6a0bef67c7b8'}, '50_floppy': {'md5': 'c68fb91d44105c8b6f2d80566314027a'}}, 'after_device_destroy': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_vm_destroy': {'50_promisc': {'md5': 'dae0434ea8ca2492422a6fde5cc47a1e'}}, 'before_vm_start': {'50_sriov': {'md5': '3ebc60cd2e4eb089820102285fad7c45'}, '50_vmfex': {'md5': '9f5abb892ddb6b3daa779985d38d9f55'}, '50_qos': {'md5': 'cdf6f89d277a886691812ce1ec2fe105'}, '50_scratchpad': {'md5': '178171eb62ba95fc548e2450013f5ad5'}, '50_numa': {'md5': '5008c2826714ac5b63748780aabd2f25'}, '50_floppy': {'md5': '202fe18705a7d4c50c40c126e8f8dbe8'}, '50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}, '50_pincpu': {'md5': '0b5fb99ff0e7acb9ad534b87c02c59e3'}, '50_hugepages': {'md5': '6713d703f9d316f1664145b700cd7904'}, '50_fileinject': {'md5': '290dafd45601e010e0f496fa5b889a76'}, '50_macspoof': {'md5': '25deea559772719b90117f2e1009d823'}, '60_checkimages': {'md5': 'abd32c723a90ab2816e2163f5e02faf7'}, '10_faqemu': {'md5': 'e3cb338be6b9075a04bed0e7400ab7e2'}, '50_directlun': {'md5': 'dba5d80fdb7119ee0f4a40c26eac8448'}, '50_smbios': {'md5': '1a6bd8eddc3a86943e558bc4d3e73bd6'}, '50_vmdisk': {'md5': '67617c6e2d796388ddc6c065a97f714e'}, '50_hostusb': {'md5': '21c58568f3eae4f0d091005e0eff891a'}, '50_isolatedprivatevlan': {'md5': '0a25d7cd0b807ea5e0dfc3318ee98c91'}}, 'after_device_migrate_destination': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'after_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_vm_migrate_destination': {'50_vmfex': {'md5': 'eac2633f47fb309b4a5b7fe76531baec'}, '50_hugepages': {'md5': '6ddc484746297308b9f9acc5fcabaf10'}, '50_directlun': {'md5': '8a29ce41de079630cb58d48e280d55c2'}}, 'after_nic_hotunplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_vm_destroy': {'50_sriov': {'md5': '5890b2899d450495a62f3611fad3c526'}, '50_directlun': {'md5': '6d05c19e68a141eeb7ee5981d68c0cfa'}, '50_hostusb': {'md5': '0a37bf0a26546e3ab5e424b13653ce2b'}, '50_hugepages': {'md5': '4a980b78f8ee908bf2c61ea9bf156f4a'}, '50_scratchpad': {'md5': '316d21a669f97e83b485b45ab0b23eee'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0', 'addr': '172.16.10.1', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'bridged': False, 'interface': u'bond0', 'ipv6gateway': '::', 'gateway': '172.16.0.1'}, 'Blue': {'iface': 'Blue', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.6']}, 'Infrastructure': {'iface': 'Infrastructure', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.4']}, 'migration': {'iface': u'bond0.5', 'addr': '', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'bridged': False, 'interface': u'bond0.5', 'ipv6gateway': '::', 'gateway': ''}, 'Red': {'iface': 'Red', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.3']}}, 'bridges': {'Blue': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.6']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'Infrastructure': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.4']}, 'Red': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.3']}}, 'uuid': '4C4C4544-0059-5310-8031-B9C04F445A31', 'lastClientIface': 'bond0', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:5F', 'MASTER': 'bond0', 'DEVICE': 'em1', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:5F', 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:60', 'MASTER': 'bond0', 'DEVICE': 'em2', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:60', 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '12', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,lahf_lm,ida,arat,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:93188a588127', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '255.255.0.0', 'addr': '172.16.10.1', 'slaves': ['em1', 'em2'], 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.10.1', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond0', 'MTU': '1500', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32057', 'cpuSpeed': '1900.070', 'version_name': 'Snow Man', 'vlans': {'bond0.3': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Red', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 3, 'mtu': '1500'}, 'bond0.4': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Infrastructure', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 4, 'mtu': '1500'}, 'bond0.5': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'DEVICE': 'bond0.5', 'NM_CONTROLLED': 'no', 'VLAN': 'yes', 'DEFROUTE': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 5, 'mtu': '1500'}, 'bond0.6': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Blue', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.6', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 6, 'mtu': '1500'}}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '24', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.10.1'}}
>Thread-14::DEBUG::2014-04-11 22:53:39,697::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getHardwareInfo with () {}
>Thread-14::DEBUG::2014-04-11 22:53:39,699::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R420', 'systemSerialNumber': '9YS1DZ1', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0059-5310-8031-B9C04F445A31', 'systemManufacturer': 'Dell Inc.'}}
>Thread-14::DEBUG::2014-04-11 22:53:39,894::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [7108a818]
>Thread-14::DEBUG::2014-04-11 22:53:39,895::task::595::TaskManager.Task::(_updateState) Task=`c0c94951-72b1-433c-9dfe-df12b3e00e2d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:53:39,895::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', conList=[{'connection': 'ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c', 'port': ''}, {'connection': 'ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/exports', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54', 'port': ''}, {'connection': 'ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'deb06d0b-14c6-490e-8fbb-552861511109', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/infrastructure', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac', 'port': ''}], options=None)
>Thread-14::WARNING::2014-04-11 22:53:39,906::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage already exists
>Thread-14::DEBUG::2014-04-11 22:53:39,907::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh02.suprtekstic.com:/var/lib/exports/storage /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage' (cwd None)
>Thread-15::DEBUG::2014-04-11 22:54:06,246::API::1134::vds::(fenceNode) fenceNode(addr=172.16.10.4,port=,agent=ipmilan,user=root,passwd=XXXX,action=status,secure=,options=lanplus)
>Thread-15::DEBUG::2014-04-11 22:54:06,337::API::1160::vds::(fenceNode) rc 0 in agent=fence_ipmilan
>ipaddr=172.16.10.4
>login=root
>action=status
>passwd=XXXX
>lanplus out Getting status of IPMI:172.16.10.4...Chassis power = On
>Done
> err 
>Thread-15::DEBUG::2014-04-11 22:54:06,389::API::1134::vds::(fenceNode) fenceNode(addr=172.16.10.4,port=,agent=ipmilan,user=root,passwd=XXXX,action=status,secure=,options=lanplus)
>Thread-15::DEBUG::2014-04-11 22:54:06,445::API::1160::vds::(fenceNode) rc 0 in agent=fence_ipmilan
>ipaddr=172.16.10.4
>login=root
>action=status
>passwd=XXXX
>lanplus out Getting status of IPMI:172.16.10.4...Chassis power = On
>Done
> err 
>Thread-14::DEBUG::2014-04-11 22:54:44,016::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage
>Thread-14::DEBUG::2014-04-11 22:54:44,037::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',)
>Thread-14::DEBUG::2014-04-11 22:54:44,038::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain}
>Thread-14::WARNING::2014-04-11 22:54:44,044::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage already exists
>Thread-14::DEBUG::2014-04-11 22:54:44,045::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh03.suprtekstic.com:/var/lib/exports/storage /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:54:44,271::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage
>Thread-14::DEBUG::2014-04-11 22:54:44,286::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',)
>Thread-14::DEBUG::2014-04-11 22:54:44,288::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain}
>Thread-14::DEBUG::2014-04-11 22:54:44,331::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/exports /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_exports' (cwd None)
>Thread-14::ERROR::2014-04-11 22:55:36,659::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-14::ERROR::2014-04-11 22:55:36,705::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-14::DEBUG::2014-04-11 22:55:36,705::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain}
>Thread-14::WARNING::2014-04-11 22:55:36,713::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage already exists
>Thread-14::DEBUG::2014-04-11 22:55:36,714::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh04.suprtekstic.com:/var/lib/exports/storage /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:55:36,957::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage
>Thread-14::DEBUG::2014-04-11 22:55:36,969::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',)
>Thread-14::DEBUG::2014-04-11 22:55:36,969::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain}
>Thread-14::DEBUG::2014-04-11 22:55:36,977::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/infrastructure /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure' (cwd None)
>Thread-14::ERROR::2014-04-11 22:56:29,307::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
>    self._mount.mount(self.options, self._vfsType)
>  File "/usr/share/vdsm/storage/mount.py", line 222, in mount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-14::ERROR::2014-04-11 22:56:29,309::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer
>    conObj.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
>    return self._mountCon.connect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
>    raise e
>MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n')
>Thread-14::DEBUG::2014-04-11 22:56:29,309::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain}
>Thread-14::WARNING::2014-04-11 22:56:29,317::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso already exists
>Thread-14::DEBUG::2014-04-11 22:56:29,317::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh01.suprtekstic.com:/var/lib/exports/iso /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:56:29,391::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso
>Thread-14::DEBUG::2014-04-11 22:56:29,394::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',)
>Thread-14::DEBUG::2014-04-11 22:56:29,395::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain}
>Thread-14::WARNING::2014-04-11 22:56:29,403::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage already exists
>Thread-14::DEBUG::2014-04-11 22:56:29,404::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh01.suprtekstic.com:/var/lib/exports/storage /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:56:29,461::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage
>Thread-14::DEBUG::2014-04-11 22:56:29,465::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('21619c8e-99ea-4813-be02-d708971e5393',)
>Thread-14::DEBUG::2014-04-11 22:56:29,465::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain}
>Thread-14::INFO::2014-04-11 22:56:29,466::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]}
>Thread-14::DEBUG::2014-04-11 22:56:29,466::task::1185::TaskManager.Task::(prepare) Task=`c0c94951-72b1-433c-9dfe-df12b3e00e2d`::finished: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]}
>Thread-14::DEBUG::2014-04-11 22:56:29,467::task::595::TaskManager.Task::(_updateState) Task=`c0c94951-72b1-433c-9dfe-df12b3e00e2d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:56:29,467::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:56:29,468::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:56:29,468::task::990::TaskManager.Task::(_decref) Task=`c0c94951-72b1-433c-9dfe-df12b3e00e2d`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:56:29,627::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:56:29,628::task::595::TaskManager.Task::(_updateState) Task=`106510a9-25a0-4951-83f3-95def0ec5828`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:56:29,628::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=38, domainsMap=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:56:29,629::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`02bb6b22-94ec-4909-8039-766ead921575`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:56:29,630::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:56:29,630::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:56:29,631::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`02bb6b22-94ec-4909-8039-766ead921575`::Granted request
>Thread-14::DEBUG::2014-04-11 22:56:29,632::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`526636e4-6be0-490c-8ae6-3155207ed1fd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool'
>Thread-14::DEBUG::2014-04-11 22:56:29,632::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:56:29,633::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:56:29,633::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`526636e4-6be0-490c-8ae6-3155207ed1fd`::Granted request
>Thread-14::INFO::2014-04-11 22:56:29,634::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7fc3ec0b4ea8
>Thread-14::INFO::2014-04-11 22:56:29,634::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 38)
>Thread-14::DEBUG::2014-04-11 22:56:29,635::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,635::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,636::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,636::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,636::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,637::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,637::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 22:56:29,638::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:56:29,638::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 22:56:29,638::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:56:29,639::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 22:56:29,639::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:56:29,667::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:56:29,668::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:56:29,822::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:56:29,823::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,824::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,824::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,824::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,825::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,825::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 22:56:29,826::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 22:56:30,149::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::DEBUG::2014-04-11 22:56:30,434::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-14::DEBUG::2014-04-11 22:56:30,459::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>Thread-14::DEBUG::2014-04-11 22:56:30,476::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-14::DEBUG::2014-04-11 22:56:30,477::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS'
>Thread-14::DEBUG::2014-04-11 22:56:30,488::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS'
>Thread-14::DEBUG::2014-04-11 22:56:30,489::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38
>Thread-14::DEBUG::2014-04-11 22:56:30,489::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>Thread-14::DEBUG::2014-04-11 22:56:30,490::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 22:56:30,505::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>Thread-14::INFO::2014-04-11 22:56:30,506::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-22::DEBUG::2014-04-11 22:56:30,507::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-14::DEBUG::2014-04-11 22:56:30,507::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `423b56b3-3148-468c-8c28-4c96fddbc1c8`
>Thread-14::INFO::2014-04-11 22:56:30,509::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-24::DEBUG::2014-04-11 22:56:30,511::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-14::DEBUG::2014-04-11 22:56:30,511::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87`
>Thread-14::INFO::2014-04-11 22:56:30,514::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-14::DEBUG::2014-04-11 22:56:30,517::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `b7663d70-e658-41fa-b9f0-8da83c9eddce`
>Thread-14::INFO::2014-04-11 22:56:30,518::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-29::DEBUG::2014-04-11 22:56:30,520::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-14::DEBUG::2014-04-11 22:56:30,528::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `3406665e-4adc-4fd4-aa1e-037547b29adb`
>Thread-33::DEBUG::2014-04-11 22:56:30,529::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-14::INFO::2014-04-11 22:56:30,529::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-33::ERROR::2014-04-11 22:56:30,531::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:56:30,536::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:56:30,539::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2014-04-11 22:56:30,541::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-39::DEBUG::2014-04-11 22:56:30,541::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-14::DEBUG::2014-04-11 22:56:30,543::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `f3b51811-4a7f-43af-8633-322b3db23c48`
>Thread-14::INFO::2014-04-11 22:56:30,549::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-39::ERROR::2014-04-11 22:56:30,557::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-43::DEBUG::2014-04-11 22:56:30,563::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-39::ERROR::2014-04-11 22:56:30,563::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-14::DEBUG::2014-04-11 22:56:30,565::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `a033c2ac-0d01-490c-9552-99ca53d6a64a`
>Thread-24::DEBUG::2014-04-11 22:56:30,568::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-39::DEBUG::2014-04-11 22:56:30,572::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-14::INFO::2014-04-11 22:56:30,574::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 21619c8e-99ea-4813-be02-d708971e5393
>Thread-14::DEBUG::2014-04-11 22:56:30,593::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `21619c8e-99ea-4813-be02-d708971e5393`
>Thread-46::DEBUG::2014-04-11 22:56:30,594::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 21619c8e-99ea-4813-be02-d708971e5393
>Thread-14::DEBUG::2014-04-11 22:56:30,599::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-33::DEBUG::2014-04-11 22:56:30,602::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:56:30,605::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-39::DEBUG::2014-04-11 22:56:30,606::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:56:30,650::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>Thread-14::DEBUG::2014-04-11 22:56:30,657::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-14::DEBUG::2014-04-11 22:56:30,660::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8 e2ffd939-380e-4fbb-8a23-5a0e69c84a87 b7663d70-e658-41fa-b9f0-8da83c9eddce 3406665e-4adc-4fd4-aa1e-037547b29adb f3b51811-4a7f-43af-8633-322b3db23c48 21619c8e-99ea-4813-be02-d708971e5393' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:56:30,696::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393
>Thread-43::DEBUG::2014-04-11 22:56:30,696::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 22:56:30,735::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000491518 s, 1.5 MB/s\n'; <rc> = 0
>Thread-43::DEBUG::2014-04-11 22:56:30,745::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-43::DEBUG::2014-04-11 22:56:30,757::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain a033c2ac-0d01-490c-9552-99ca53d6a64a changed its status to Valid
>Thread-43::INFO::2014-04-11 22:56:30,758::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-53::DEBUG::2014-04-11 22:56:30,758::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-43::DEBUG::2014-04-11 22:56:30,761::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-53::DEBUG::2014-04-11 22:56:30,762::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-53::DEBUG::2014-04-11 22:56:30,789::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-54::INFO::2014-04-11 22:56:30,790::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-22::DEBUG::2014-04-11 22:56:30,843::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-24::DEBUG::2014-04-11 22:56:30,868::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-22::DEBUG::2014-04-11 22:56:30,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955']
>Thread-24::DEBUG::2014-04-11 22:56:30,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_Storage', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=e2ffd939-380e-4fbb-8a23-5a0e69c84a87', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=aa653ed432ff22d61d7b0ce5019a22c0a8bb40dd']
>Thread-22::DEBUG::2014-04-11 22:56:30,889::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-24::DEBUG::2014-04-11 22:56:30,890::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-22::DEBUG::2014-04-11 22:56:30,893::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '423b56b3-3148-468c-8c28-4c96fddbc1c8_imageNS'
>Thread-24::DEBUG::2014-04-11 22:56:30,895::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87_imageNS'
>Thread-22::DEBUG::2014-04-11 22:56:30,897::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '423b56b3-3148-468c-8c28-4c96fddbc1c8_volumeNS'
>Thread-24::DEBUG::2014-04-11 22:56:30,898::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87_volumeNS'
>Thread-29::DEBUG::2014-04-11 22:56:30,917::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-33::DEBUG::2014-04-11 22:56:30,979::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-39::DEBUG::2014-04-11 22:56:30,983::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-29::DEBUG::2014-04-11 22:56:30,991::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40']
>Thread-39::WARNING::2014-04-11 22:56:31,015::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-33::WARNING::2014-04-11 22:56:31,015::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-46::DEBUG::2014-04-11 22:56:31,016::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-14::DEBUG::2014-04-11 22:56:31,030::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n  Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found\n  Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87\n  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n  Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found\n  Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393\n'; <rc> = 5
>Thread-29::DEBUG::2014-04-11 22:56:31,064::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-14::WARNING::2014-04-11 22:56:31,080::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', '  Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8', '  Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found', '  Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87', '  Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', '  Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce', '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb', '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48', '  Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found', '  Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393']
>Thread-29::DEBUG::2014-04-11 22:56:31,083::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'b7663d70-e658-41fa-b9f0-8da83c9eddce_imageNS'
>Thread-14::DEBUG::2014-04-11 22:56:31,109::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-29::DEBUG::2014-04-11 22:56:31,118::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'b7663d70-e658-41fa-b9f0-8da83c9eddce_volumeNS'
>Thread-46::DEBUG::2014-04-11 22:56:31,135::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2']
>Thread-39::ERROR::2014-04-11 22:56:31,355::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-46::DEBUG::2014-04-11 22:56:31,395::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-39::ERROR::2014-04-11 22:56:31,414::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-46::DEBUG::2014-04-11 22:56:31,431::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '21619c8e-99ea-4813-be02-d708971e5393_imageNS'
>Thread-14::INFO::2014-04-11 22:56:31,434::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-39::DEBUG::2014-04-11 22:56:31,445::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain f3b51811-4a7f-43af-8633-322b3db23c48 changed its status to Invalid
>Thread-14::INFO::2014-04-11 22:56:31,457::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-46::DEBUG::2014-04-11 22:56:31,467::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '21619c8e-99ea-4813-be02-d708971e5393_volumeNS'
>Thread-73::DEBUG::2014-04-11 22:56:31,467::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-14::INFO::2014-04-11 22:56:31,475::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-73::DEBUG::2014-04-11 22:56:31,490::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-14::INFO::2014-04-11 22:56:31,493::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/21619c8e-99ea-4813-be02-d708971e5393
>Thread-73::DEBUG::2014-04-11 22:56:31,514::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-14::INFO::2014-04-11 22:56:31,514::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-14::INFO::2014-04-11 22:56:31,532::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/mastersd
>Thread-14::DEBUG::2014-04-11 22:56:31,553::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 22:56:31,570::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 22:56:31,580::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 22:56:31,590::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 22:56:31,612::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-33::ERROR::2014-04-11 22:56:31,625::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:56:31,630::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-33::ERROR::2014-04-11 22:56:31,639::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:56:31,643::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-33::DEBUG::2014-04-11 22:56:31,657::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3406665e-4adc-4fd4-aa1e-037547b29adb changed its status to Invalid
>Thread-14::DEBUG::2014-04-11 22:56:31,662::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-75::DEBUG::2014-04-11 22:56:31,673::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-14::INFO::2014-04-11 22:56:31,681::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
>Thread-75::DEBUG::2014-04-11 22:56:31,685::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-14::DEBUG::2014-04-11 22:56:31,692::task::1185::TaskManager.Task::(prepare) Task=`106510a9-25a0-4951-83f3-95def0ec5828`::finished: True
>Thread-75::DEBUG::2014-04-11 22:56:31,701::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-14::DEBUG::2014-04-11 22:56:31,704::task::595::TaskManager.Task::(_updateState) Task=`106510a9-25a0-4951-83f3-95def0ec5828`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:56:31,715::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:56:31,725::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:56:31,737::task::990::TaskManager.Task::(_decref) Task=`106510a9-25a0-4951-83f3-95def0ec5828`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:56:31,826::task::595::TaskManager.Task::(_updateState) Task=`872ca3a1-de16-48aa-ba43-65a07a77546d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:56:31,838::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:56:31,869::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000491518', 'lastCheck': '1.1', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:56:31,878::task::1185::TaskManager.Task::(prepare) Task=`872ca3a1-de16-48aa-ba43-65a07a77546d`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000491518', 'lastCheck': '1.1', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:56:31,889::task::595::TaskManager.Task::(_updateState) Task=`872ca3a1-de16-48aa-ba43-65a07a77546d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:56:31,899::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:56:31,909::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:56:31,919::task::990::TaskManager.Task::(_decref) Task=`872ca3a1-de16-48aa-ba43-65a07a77546d`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:56:31,929::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:56:32,003::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:56:32,038::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000248225 s, 1.5 MB/s\n'; <rc> = 0
>Thread-24::DEBUG::2014-04-11 22:56:32,073::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 changed its status to Valid
>Thread-77::DEBUG::2014-04-11 22:56:32,083::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-77::DEBUG::2014-04-11 22:56:32,091::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-77::DEBUG::2014-04-11 22:56:32,103::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-78::INFO::2014-04-11 22:56:32,105::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-22::DEBUG::2014-04-11 22:56:32,112::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000456571 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:56:32,136::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-22::DEBUG::2014-04-11 22:56:32,145::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 changed its status to Valid
>Thread-79::DEBUG::2014-04-11 22:56:32,153::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-22::INFO::2014-04-11 22:56:32,154::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-79::DEBUG::2014-04-11 22:56:32,161::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-22::DEBUG::2014-04-11 22:56:32,164::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-79::DEBUG::2014-04-11 22:56:32,173::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-80::INFO::2014-04-11 22:56:32,175::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-29::DEBUG::2014-04-11 22:56:32,196::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:56:32,226::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:56:32,240::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000367226 s, 2.1 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:56:32,250::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-29::DEBUG::2014-04-11 22:56:32,254::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain b7663d70-e658-41fa-b9f0-8da83c9eddce changed its status to Valid
>Thread-46::DEBUG::2014-04-11 22:56:32,254::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000315664 s, 2.4 MB/s\n'; <rc> = 0
>Thread-81::DEBUG::2014-04-11 22:56:32,258::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-29::INFO::2014-04-11 22:56:32,259::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1)
>Thread-81::DEBUG::2014-04-11 22:56:32,263::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-29::DEBUG::2014-04-11 22:56:32,265::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-81::DEBUG::2014-04-11 22:56:32,272::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-82::INFO::2014-04-11 22:56:32,274::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-46::DEBUG::2014-04-11 22:56:32,282::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-46::DEBUG::2014-04-11 22:56:32,282::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 21619c8e-99ea-4813-be02-d708971e5393 changed its status to Valid
>Thread-83::DEBUG::2014-04-11 22:56:32,283::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
>Thread-46::INFO::2014-04-11 22:56:32,284::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1)
>Thread-83::DEBUG::2014-04-11 22:56:32,286::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
>Thread-46::DEBUG::2014-04-11 22:56:32,288::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-83::DEBUG::2014-04-11 22:56:32,292::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
>Thread-84::INFO::2014-04-11 22:56:32,293::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
>Thread-14::DEBUG::2014-04-11 22:56:33,094::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call setMOMPolicyParameters with ({'balloonEnabled': True, 'ksmEnabled': True},) {} flowID [7108a818]
>Thread-14::DEBUG::2014-04-11 22:56:33,237::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
>Thread-43::DEBUG::2014-04-11 22:56:40,819::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 22:56:40,836::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000699507 s, 1.1 MB/s\n'; <rc> = 0
>Thread-43::DEBUG::2014-04-11 22:56:40,850::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-43::INFO::2014-04-11 22:56:40,852::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-43::DEBUG::2014-04-11 22:56:40,852::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-39::ERROR::2014-04-11 22:56:41,468::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:56:41,469::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:56:41,469::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:56:41,470::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:56:41,620::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:56:41,621::task::595::TaskManager.Task::(_updateState) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:56:41,622::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:56:41,623::task::866::TaskManager.Task::(_setError) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus
>    status = self._getSpmStatusInfo(pool)
>  File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo
>    (pool.spmRole,) + pool.getSpmStatus()))
>  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
>    return self._backend.getSpmStatus()
>  File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus
>    lVer, spmId = self.masterDomain.inquireClusterLock()
>  File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock
>    return self._clusterLock.inquire()
>  File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire
>    resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET)
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:56:41,650::task::885::TaskManager.Task::(_run) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::Task._run: 3345588b-6ade-4347-a769-d92a3d1c3fb8 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:56:41,650::task::1211::TaskManager.Task::(stop) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:56:41,650::task::990::TaskManager.Task::(_decref) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:56:41,651::task::1168::TaskManager.Task::(prepare) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100
>Thread-14::DEBUG::2014-04-11 22:56:41,651::task::1173::TaskManager.Task::(prepare) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:56:41,652::task::990::TaskManager.Task::(_decref) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:56:41,652::task::925::TaskManager.Task::(_doAbort) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:56:41,652::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:56:41,653::task::595::TaskManager.Task::(_updateState) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:56:41,653::task::550::TaskManager.Task::(__state_aborting) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:56:41,653::task::595::TaskManager.Task::(_updateState) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:56:41,653::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:56:41,654::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:56:41,654::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-33::ERROR::2014-04-11 22:56:41,674::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:56:41,674::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:56:41,675::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:56:41,676::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:56:41,841::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:56:41,844::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:56:41,880::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:56:41,880::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:56:41,954::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:56:41,955::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:56:41,956::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:56:41,985::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:56:41,986::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-24::DEBUG::2014-04-11 22:56:42,109::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:56:42,122::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000360783 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:56:42,201::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:56:42,215::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000629432 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:56:42,218::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-22::INFO::2014-04-11 22:56:42,218::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-22::DEBUG::2014-04-11 22:56:42,219::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-29::DEBUG::2014-04-11 22:56:42,353::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:56:42,380::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000417906 s, 1.8 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:56:42,385::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-29::INFO::2014-04-11 22:56:42,388::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1)
>Thread-29::DEBUG::2014-04-11 22:56:42,389::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-46::DEBUG::2014-04-11 22:56:42,395::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:56:42,407::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000313101 s, 2.4 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:56:42,411::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-46::INFO::2014-04-11 22:56:42,412::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1)
>Thread-46::DEBUG::2014-04-11 22:56:42,412::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-14::DEBUG::2014-04-11 22:56:48,555::task::595::TaskManager.Task::(_updateState) Task=`25a0efc7-e8c5-40db-8eda-6880ae3b6515`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:56:48,563::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:56:48,563::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000629432', 'lastCheck': '6.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000360783', 'lastCheck': '6.4', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000417906', 'lastCheck': '6.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.7', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000699507', 'lastCheck': '7.7', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000313101', 'lastCheck': '6.2', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:56:48,563::task::1185::TaskManager.Task::(prepare) Task=`25a0efc7-e8c5-40db-8eda-6880ae3b6515`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000629432', 'lastCheck': '6.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000360783', 'lastCheck': '6.4', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000417906', 'lastCheck': '6.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.7', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000699507', 'lastCheck': '7.7', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000313101', 'lastCheck': '6.2', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:56:48,563::task::595::TaskManager.Task::(_updateState) Task=`25a0efc7-e8c5-40db-8eda-6880ae3b6515`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:56:48,564::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:56:48,564::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:56:48,564::task::990::TaskManager.Task::(_decref) Task=`25a0efc7-e8c5-40db-8eda-6880ae3b6515`::ref 0 aborting False
>Thread-43::DEBUG::2014-04-11 22:56:50,876::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 22:56:50,890::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000607005 s, 1.2 MB/s\n'; <rc> = 0
>Thread-43::DEBUG::2014-04-11 22:56:50,894::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-43::INFO::2014-04-11 22:56:50,895::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-43::DEBUG::2014-04-11 22:56:50,896::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-14::DEBUG::2014-04-11 22:56:51,787::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:56:51,788::task::595::TaskManager.Task::(_updateState) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:56:51,789::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:56:51,790::task::866::TaskManager.Task::(_setError) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus
>    status = self._getSpmStatusInfo(pool)
>  File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo
>    (pool.spmRole,) + pool.getSpmStatus()))
>  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
>    return self._backend.getSpmStatus()
>  File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus
>    lVer, spmId = self.masterDomain.inquireClusterLock()
>  File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock
>    return self._clusterLock.inquire()
>  File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire
>    resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET)
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:56:51,791::task::885::TaskManager.Task::(_run) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::Task._run: efe027f5-864e-4f41-b194-04d53c46e9e6 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:56:51,791::task::1211::TaskManager.Task::(stop) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:56:51,791::task::990::TaskManager.Task::(_decref) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:56:51,792::task::1168::TaskManager.Task::(prepare) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100
>Thread-14::DEBUG::2014-04-11 22:56:51,792::task::1173::TaskManager.Task::(prepare) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:56:51,793::task::990::TaskManager.Task::(_decref) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:56:51,793::task::925::TaskManager.Task::(_doAbort) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:56:51,793::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:56:51,794::task::595::TaskManager.Task::(_updateState) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:56:51,794::task::550::TaskManager.Task::(__state_aborting) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:56:51,794::task::595::TaskManager.Task::(_updateState) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:56:51,794::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:56:51,795::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:56:51,795::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-39::ERROR::2014-04-11 22:56:51,881::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:56:51,882::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:56:51,882::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:56:51,883::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:56:51,987::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:56:51,988::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:56:51,988::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:56:51,990::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:56:52,164::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:56:52,181::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:56:52,192::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-24::DEBUG::2014-04-11 22:56:52,203::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.00030025 s, 1.2 MB/s\n'; <rc> = 0
>Thread-39::ERROR::2014-04-11 22:56:52,255::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:56:52,260::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:56:52,276::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:56:52,287::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:56:52,291::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:56:52,341::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:56:52,344::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-22::DEBUG::2014-04-11 22:56:52,363::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:56:52,376::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000535976 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:56:52,379::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-22::INFO::2014-04-11 22:56:52,380::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-22::DEBUG::2014-04-11 22:56:52,380::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-29::DEBUG::2014-04-11 22:56:52,441::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:56:52,479::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000497721 s, 1.5 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:56:52,491::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-29::INFO::2014-04-11 22:56:52,493::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1)
>Thread-29::DEBUG::2014-04-11 22:56:52,495::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-46::DEBUG::2014-04-11 22:56:52,512::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:56:52,523::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000349473 s, 2.2 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:56:52,528::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-46::INFO::2014-04-11 22:56:52,528::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1)
>Thread-46::DEBUG::2014-04-11 22:56:52,529::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-43::DEBUG::2014-04-11 22:57:00,917::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 22:57:00,931::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000515515 s, 1.5 MB/s\n'; <rc> = 0
>Thread-43::DEBUG::2014-04-11 22:57:00,936::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-43::INFO::2014-04-11 22:57:00,937::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-43::DEBUG::2014-04-11 22:57:00,937::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-14::DEBUG::2014-04-11 22:57:01,917::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:01,918::task::595::TaskManager.Task::(_updateState) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:01,919::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:57:01,920::task::866::TaskManager.Task::(_setError) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus
>    status = self._getSpmStatusInfo(pool)
>  File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo
>    (pool.spmRole,) + pool.getSpmStatus()))
>  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
>    return self._backend.getSpmStatus()
>  File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus
>    lVer, spmId = self.masterDomain.inquireClusterLock()
>  File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock
>    return self._clusterLock.inquire()
>  File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire
>    resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET)
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:57:01,921::task::885::TaskManager.Task::(_run) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::Task._run: fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:57:01,921::task::1211::TaskManager.Task::(stop) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:57:01,921::task::990::TaskManager.Task::(_decref) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:57:01,922::task::1168::TaskManager.Task::(prepare) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100
>Thread-14::DEBUG::2014-04-11 22:57:01,922::task::1173::TaskManager.Task::(prepare) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:57:01,923::task::990::TaskManager.Task::(_decref) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:57:01,923::task::925::TaskManager.Task::(_doAbort) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:57:01,923::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:01,923::task::595::TaskManager.Task::(_updateState) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:57:01,924::task::550::TaskManager.Task::(__state_aborting) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:57:01,924::task::595::TaskManager.Task::(_updateState) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:57:01,924::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:01,925::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:57:01,925::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-39::ERROR::2014-04-11 22:57:02,268::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:57:02,270::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:57:02,274::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:57:02,290::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:57:02,309::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:57:02,321::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000338993 s, 1.1 MB/s\n'; <rc> = 0
>Thread-33::ERROR::2014-04-11 22:57:02,348::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:57:02,348::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:57:02,349::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:57:02,350::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:57:02,406::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:57:02,420::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000613988 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:57:02,422::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-22::INFO::2014-04-11 22:57:02,423::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-22::DEBUG::2014-04-11 22:57:02,424::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-29::DEBUG::2014-04-11 22:57:02,525::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:57:02,560::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00051581 s, 1.5 MB/s\n'; <rc> = 0
>Thread-39::DEBUG::2014-04-11 22:57:02,579::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-29::DEBUG::2014-04-11 22:57:02,579::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-29::INFO::2014-04-11 22:57:02,592::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1)
>Thread-39::WARNING::2014-04-11 22:57:02,598::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-29::DEBUG::2014-04-11 22:57:02,599::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-33::DEBUG::2014-04-11 22:57:02,604::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:57:02,633::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:57:02,656::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-39::ERROR::2014-04-11 22:57:02,678::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:57:02,684::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::ERROR::2014-04-11 22:57:02,725::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:57:02,729::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-46::DEBUG::2014-04-11 22:57:02,748::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:57:02,761::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000388102 s, 1.9 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:57:02,783::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-46::INFO::2014-04-11 22:57:02,784::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1)
>Thread-46::DEBUG::2014-04-11 22:57:02,784::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-14::DEBUG::2014-04-11 22:57:03,876::task::595::TaskManager.Task::(_updateState) Task=`c7b09696-4f2c-4c89-a29d-3c2fe150ecfc`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:03,877::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:57:03,878::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000613988', 'lastCheck': '1.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338993', 'lastCheck': '1.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00051581', 'lastCheck': '1.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000515515', 'lastCheck': '2.9', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000388102', 'lastCheck': '1.1', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:57:03,878::task::1185::TaskManager.Task::(prepare) Task=`c7b09696-4f2c-4c89-a29d-3c2fe150ecfc`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000613988', 'lastCheck': '1.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338993', 'lastCheck': '1.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00051581', 'lastCheck': '1.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000515515', 'lastCheck': '2.9', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000388102', 'lastCheck': '1.1', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:57:03,878::task::595::TaskManager.Task::(_updateState) Task=`c7b09696-4f2c-4c89-a29d-3c2fe150ecfc`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:03,879::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:03,879::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:03,879::task::990::TaskManager.Task::(_decref) Task=`c7b09696-4f2c-4c89-a29d-3c2fe150ecfc`::ref 0 aborting False
>Thread-43::DEBUG::2014-04-11 22:57:10,958::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 22:57:10,971::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000523507 s, 1.4 MB/s\n'; <rc> = 0
>Thread-43::DEBUG::2014-04-11 22:57:10,980::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-43::INFO::2014-04-11 22:57:10,981::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-43::DEBUG::2014-04-11 22:57:10,982::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-14::DEBUG::2014-04-11 22:57:12,046::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:12,047::task::595::TaskManager.Task::(_updateState) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:12,048::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:57:12,049::task::866::TaskManager.Task::(_setError) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus
>    status = self._getSpmStatusInfo(pool)
>  File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo
>    (pool.spmRole,) + pool.getSpmStatus()))
>  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
>    return self._backend.getSpmStatus()
>  File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus
>    lVer, spmId = self.masterDomain.inquireClusterLock()
>  File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock
>    return self._clusterLock.inquire()
>  File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire
>    resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET)
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:57:12,050::task::885::TaskManager.Task::(_run) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::Task._run: 14d97fba-8715-4b83-ae07-0ff44c69e520 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:57:12,050::task::1211::TaskManager.Task::(stop) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:57:12,050::task::990::TaskManager.Task::(_decref) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:57:12,051::task::1168::TaskManager.Task::(prepare) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100
>Thread-14::DEBUG::2014-04-11 22:57:12,051::task::1173::TaskManager.Task::(prepare) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:57:12,051::task::990::TaskManager.Task::(_decref) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:57:12,052::task::925::TaskManager.Task::(_doAbort) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:57:12,052::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:12,052::task::595::TaskManager.Task::(_updateState) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:57:12,053::task::550::TaskManager.Task::(__state_aborting) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:57:12,053::task::595::TaskManager.Task::(_updateState) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:57:12,053::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:12,054::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:57:12,054::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-24::DEBUG::2014-04-11 22:57:12,349::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:57:12,361::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000321341 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:57:12,445::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:57:12,458::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000486723 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:57:12,461::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-22::INFO::2014-04-11 22:57:12,462::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-22::DEBUG::2014-04-11 22:57:12,462::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-29::DEBUG::2014-04-11 22:57:12,636::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:57:12,648::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000415012 s, 1.8 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:57:12,651::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-29::INFO::2014-04-11 22:57:12,651::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1)
>Thread-29::DEBUG::2014-04-11 22:57:12,652::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-39::ERROR::2014-04-11 22:57:12,698::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:57:12,698::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:57:12,699::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:57:12,700::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:57:12,736::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:57:12,736::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:57:12,737::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:57:12,738::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:57:12,811::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:57:12,824::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000351933 s, 2.1 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:57:12,828::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-46::INFO::2014-04-11 22:57:12,829::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1)
>Thread-46::DEBUG::2014-04-11 22:57:12,829::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-39::DEBUG::2014-04-11 22:57:12,966::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:57:12,968::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:57:13,003::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:57:13,003::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:57:13,010::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:57:13,012::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:57:13,012::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:57:13,042::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:57:13,042::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:57:19,200::task::595::TaskManager.Task::(_updateState) Task=`51f35ac7-1d14-473e-9106-232fe32ed366`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:19,215::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:57:19,216::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000486723', 'lastCheck': '6.8', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000321341', 'lastCheck': '6.9', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000415012', 'lastCheck': '6.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000523507', 'lastCheck': '8.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000351933', 'lastCheck': '6.4', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:57:19,216::task::1185::TaskManager.Task::(prepare) Task=`51f35ac7-1d14-473e-9106-232fe32ed366`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000486723', 'lastCheck': '6.8', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000321341', 'lastCheck': '6.9', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000415012', 'lastCheck': '6.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000523507', 'lastCheck': '8.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000351933', 'lastCheck': '6.4', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:57:19,217::task::595::TaskManager.Task::(_updateState) Task=`51f35ac7-1d14-473e-9106-232fe32ed366`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:19,217::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:19,217::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:19,218::task::990::TaskManager.Task::(_decref) Task=`51f35ac7-1d14-473e-9106-232fe32ed366`::ref 0 aborting False
>Thread-43::DEBUG::2014-04-11 22:57:21,008::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 22:57:21,022::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000467351 s, 1.6 MB/s\n'; <rc> = 0
>Thread-43::DEBUG::2014-04-11 22:57:21,039::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-43::INFO::2014-04-11 22:57:21,040::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-43::DEBUG::2014-04-11 22:57:21,040::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-14::DEBUG::2014-04-11 22:57:22,188::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:22,189::task::595::TaskManager.Task::(_updateState) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:22,190::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::ERROR::2014-04-11 22:57:22,191::task::866::TaskManager.Task::(_setError) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus
>    status = self._getSpmStatusInfo(pool)
>  File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo
>    (pool.spmRole,) + pool.getSpmStatus()))
>  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
>    return self._backend.getSpmStatus()
>  File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus
>    lVer, spmId = self.masterDomain.inquireClusterLock()
>  File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock
>    return self._clusterLock.inquire()
>  File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire
>    resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET)
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:57:22,193::task::885::TaskManager.Task::(_run) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::Task._run: ac3650a0-f5f8-4c65-b742-e802954f5d16 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 22:57:22,194::task::1211::TaskManager.Task::(stop) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 22:57:22,194::task::990::TaskManager.Task::(_decref) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 22:57:22,194::task::1168::TaskManager.Task::(prepare) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100
>Thread-14::DEBUG::2014-04-11 22:57:22,195::task::1173::TaskManager.Task::(prepare) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-14::DEBUG::2014-04-11 22:57:22,195::task::990::TaskManager.Task::(_decref) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 22:57:22,195::task::925::TaskManager.Task::(_doAbort) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 22:57:22,196::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:22,196::task::595::TaskManager.Task::(_updateState) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 22:57:22,196::task::550::TaskManager.Task::(__state_aborting) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 22:57:22,197::task::595::TaskManager.Task::(_updateState) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 22:57:22,197::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:22,197::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 22:57:22,198::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused')
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SanlockException: (111, 'Sanlock resource read failure', 'Connection refused')
>Thread-24::DEBUG::2014-04-11 22:57:22,391::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:57:22,403::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000368864 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:57:22,488::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:57:22,501::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000461848 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:57:22,504::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-22::INFO::2014-04-11 22:57:22,505::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-22::DEBUG::2014-04-11 22:57:22,505::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-29::DEBUG::2014-04-11 22:57:22,677::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:57:22,689::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000441191 s, 1.7 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:57:22,692::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-29::INFO::2014-04-11 22:57:22,692::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1)
>Thread-29::DEBUG::2014-04-11 22:57:22,693::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-46::DEBUG::2014-04-11 22:57:22,855::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:57:22,866::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000328371 s, 2.3 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:57:22,870::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId
>    hostId, self._idsPath)
>SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused')
>Thread-46::INFO::2014-04-11 22:57:22,871::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1)
>Thread-46::DEBUG::2014-04-11 22:57:22,872::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain
>    self.domain.acquireHostId(self.hostId, async=True)
>  File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId
>    self._clusterLock.acquireHostId(hostId, async)
>  File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId
>    raise se.AcquireHostIdFailure(self._sdUUID, e)
>AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused'))
>Thread-39::ERROR::2014-04-11 22:57:23,004::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:57:23,005::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:57:23,005::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:57:23,007::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:57:23,044::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:57:23,044::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:57:23,045::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:57:23,046::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:57:23,311::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:57:23,313::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-33::DEBUG::2014-04-11 22:57:23,353::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-39::ERROR::2014-04-11 22:57:23,355::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:57:23,360::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::WARNING::2014-04-11 22:57:23,364::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:57:23,365::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:57:23,395::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:57:23,395::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-43::DEBUG::2014-04-11 22:57:31,064::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 22:57:31,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000570109 s, 1.3 MB/s\n'; <rc> = 0
>Thread-43::INFO::2014-04-11 22:57:31,086::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-43::DEBUG::2014-04-11 22:57:31,086::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>Thread-14::DEBUG::2014-04-11 22:57:32,314::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:32,315::task::595::TaskManager.Task::(_updateState) Task=`5b62f416-800f-4558-8185-3a1e86db2c53`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:32,316::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 22:57:32,351::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 22:57:32,352::task::1185::TaskManager.Task::(prepare) Task=`5b62f416-800f-4558-8185-3a1e86db2c53`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 22:57:32,352::task::595::TaskManager.Task::(_updateState) Task=`5b62f416-800f-4558-8185-3a1e86db2c53`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:32,352::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:32,353::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:32,353::task::990::TaskManager.Task::(_decref) Task=`5b62f416-800f-4558-8185-3a1e86db2c53`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:32,418::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:32,423::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:32,427::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
>Thread-14::DEBUG::2014-04-11 22:57:32,436::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`6baf763a-af73-41f5-865b-9c7a96010385`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart'
>Thread-14::DEBUG::2014-04-11 22:57:32,439::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 22:57:32,444::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 22:57:32,447::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`6baf763a-af73-41f5-865b-9c7a96010385`::Granted request
>Thread-14::DEBUG::2014-04-11 22:57:32,455::task::827::TaskManager.Task::(resourceAcquired) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive)
>Thread-14::DEBUG::2014-04-11 22:57:32,458::task::990::TaskManager.Task::(_decref) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::ref 1 aborting False
>Thread-14::INFO::2014-04-11 22:57:32,462::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
>Thread-14::DEBUG::2014-04-11 22:57:32,465::task::1180::TaskManager.Task::(prepare) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Prepare: 1 jobs exist, move to acquiring
>Thread-14::DEBUG::2014-04-11 22:57:32,469::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state preparing -> state acquiring
>Thread-14::DEBUG::2014-04-11 22:57:32,470::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state acquiring -> state queued
>Thread-14::DEBUG::2014-04-11 22:57:32,472::taskManager::50::TaskManager::(_queueTask) queuing task: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:32,476::taskManager::56::TaskManager::(_queueTask) task queued: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-5::DEBUG::2014-04-11 22:57:32,476::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
>Thread-14::DEBUG::2014-04-11 22:57:32,481::task::1182::TaskManager.Task::(prepare) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::returning
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,482::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 9d3d90f5-8732-4a54-854e-573f3fdcd2de running: <bound method Task.commit of <storage.task.Task instance at 0x7fc3f818bc68>> with: None
>Thread-14::DEBUG::2014-04-11 22:57:32,488::task::990::TaskManager.Task::(_decref) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::ref 0 aborting False
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,489::task::1194::TaskManager.Task::(commit) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::committing task: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,496::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state queued -> state running
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,503::task::905::TaskManager.Task::(_runJobs) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {})
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,510::task::331::TaskManager.Task::(run) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,517::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,523::misc::764::SamplingMethod::(__call__) Got in to sampling method
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,568::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,582::misc::772::SamplingMethod::(__call__) Returning last result
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 22:57:32,604::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-24::DEBUG::2014-04-11 22:57:32,691::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:57:32,753::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000262429 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:57:32,824::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:57:32,860::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000421615 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::INFO::2014-04-11 22:57:32,875::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-22::DEBUG::2014-04-11 22:57:32,880::clusterlock::202::SANLock::(acquireHostId) Host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 successfully acquired (id: 1)
>Thread-29::DEBUG::2014-04-11 22:57:32,950::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:57:32,983::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000441694 s, 1.7 MB/s\n'; <rc> = 0
>Thread-29::INFO::2014-04-11 22:57:32,996::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1)
>Thread-29::DEBUG::2014-04-11 22:57:33,000::clusterlock::202::SANLock::(acquireHostId) Host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce successfully acquired (id: 1)
>Thread-46::DEBUG::2014-04-11 22:57:33,010::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:57:33,022::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000240185 s, 3.1 MB/s\n'; <rc> = 0
>Thread-46::INFO::2014-04-11 22:57:33,028::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1)
>Thread-46::DEBUG::2014-04-11 22:57:33,028::clusterlock::202::SANLock::(acquireHostId) Host id for domain 21619c8e-99ea-4813-be02-d708971e5393 successfully acquired (id: 1)
>Thread-39::ERROR::2014-04-11 22:57:33,364::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:57:33,365::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:57:33,365::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:57:33,366::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:57:33,396::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:57:33,397::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:57:33,398::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:57:33,399::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:57:33,512::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:33,513::task::595::TaskManager.Task::(_updateState) Task=`f3a1968b-0e06-486c-af6a-32813115b9d1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:33,513::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:33,514::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:33,514::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:33,514::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:33,515::task::1185::TaskManager.Task::(prepare) Task=`f3a1968b-0e06-486c-af6a-32813115b9d1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:33,515::task::595::TaskManager.Task::(_updateState) Task=`f3a1968b-0e06-486c-af6a-32813115b9d1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:33,515::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:33,516::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:33,516::task::990::TaskManager.Task::(_decref) Task=`f3a1968b-0e06-486c-af6a-32813115b9d1`::ref 0 aborting False
>Thread-33::DEBUG::2014-04-11 22:57:33,685::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-39::DEBUG::2014-04-11 22:57:33,694::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:57:33,696::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-33::WARNING::2014-04-11 22:57:33,696::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:57:33,702::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-39::ERROR::2014-04-11 22:57:33,772::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:57:33,781::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::ERROR::2014-04-11 22:57:33,785::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:57:33,788::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:57:34,522::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:34,522::task::595::TaskManager.Task::(_updateState) Task=`699efc26-f273-473b-b950-ae00fe62d905`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:34,523::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:34,523::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:34,524::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:34,524::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:34,524::task::1185::TaskManager.Task::(prepare) Task=`699efc26-f273-473b-b950-ae00fe62d905`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:34,525::task::595::TaskManager.Task::(_updateState) Task=`699efc26-f273-473b-b950-ae00fe62d905`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:34,525::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:34,525::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:34,526::task::990::TaskManager.Task::(_decref) Task=`699efc26-f273-473b-b950-ae00fe62d905`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:34,537::task::595::TaskManager.Task::(_updateState) Task=`ff1d35e9-01d3-4234-9054-b2521de9ee0e`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:34,537::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:57:34,538::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000421615', 'lastCheck': '1.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000262429', 'lastCheck': '1.8', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000441694', 'lastCheck': '1.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '3.5', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000240185', 'lastCheck': '1.5', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:57:34,538::task::1185::TaskManager.Task::(prepare) Task=`ff1d35e9-01d3-4234-9054-b2521de9ee0e`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000421615', 'lastCheck': '1.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000262429', 'lastCheck': '1.8', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000441694', 'lastCheck': '1.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '3.5', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000240185', 'lastCheck': '1.5', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:57:34,539::task::595::TaskManager.Task::(_updateState) Task=`ff1d35e9-01d3-4234-9054-b2521de9ee0e`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:34,539::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:34,540::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:34,540::task::990::TaskManager.Task::(_decref) Task=`ff1d35e9-01d3-4234-9054-b2521de9ee0e`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:35,531::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:35,532::task::595::TaskManager.Task::(_updateState) Task=`b0e418ac-be48-428f-a286-c28ed9f912c3`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:35,532::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:35,533::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:35,533::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:35,533::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:35,534::task::1185::TaskManager.Task::(prepare) Task=`b0e418ac-be48-428f-a286-c28ed9f912c3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:35,534::task::595::TaskManager.Task::(_updateState) Task=`b0e418ac-be48-428f-a286-c28ed9f912c3`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:35,534::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:35,535::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:35,535::task::990::TaskManager.Task::(_decref) Task=`b0e418ac-be48-428f-a286-c28ed9f912c3`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:36,540::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:36,541::task::595::TaskManager.Task::(_updateState) Task=`f2a93513-efb6-4a68-9882-847b3db44d3f`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:36,541::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:36,541::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:36,542::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:36,542::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:36,542::task::1185::TaskManager.Task::(prepare) Task=`f2a93513-efb6-4a68-9882-847b3db44d3f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:36,543::task::595::TaskManager.Task::(_updateState) Task=`f2a93513-efb6-4a68-9882-847b3db44d3f`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:36,543::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:36,543::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:36,543::task::990::TaskManager.Task::(_decref) Task=`f2a93513-efb6-4a68-9882-847b3db44d3f`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:37,549::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:37,551::task::595::TaskManager.Task::(_updateState) Task=`08fab4f4-dd94-4891-87f9-e98849b13d4e`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:37,551::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:37,552::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:37,552::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:37,553::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:37,553::task::1185::TaskManager.Task::(prepare) Task=`08fab4f4-dd94-4891-87f9-e98849b13d4e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:37,553::task::595::TaskManager.Task::(_updateState) Task=`08fab4f4-dd94-4891-87f9-e98849b13d4e`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:37,554::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:37,554::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:37,554::task::990::TaskManager.Task::(_decref) Task=`08fab4f4-dd94-4891-87f9-e98849b13d4e`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:38,559::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:38,560::task::595::TaskManager.Task::(_updateState) Task=`9b8bf607-64e4-4d85-a77b-8caf26cebf26`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:38,561::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:38,561::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:38,561::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:38,562::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:38,562::task::1185::TaskManager.Task::(prepare) Task=`9b8bf607-64e4-4d85-a77b-8caf26cebf26`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:38,562::task::595::TaskManager.Task::(_updateState) Task=`9b8bf607-64e4-4d85-a77b-8caf26cebf26`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:38,563::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:38,563::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:38,563::task::990::TaskManager.Task::(_decref) Task=`9b8bf607-64e4-4d85-a77b-8caf26cebf26`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:39,568::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:39,568::task::595::TaskManager.Task::(_updateState) Task=`699d6680-b4a5-477d-a18a-146b95594b04`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:39,569::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:39,569::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:39,569::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:39,570::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:39,570::task::1185::TaskManager.Task::(prepare) Task=`699d6680-b4a5-477d-a18a-146b95594b04`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:39,570::task::595::TaskManager.Task::(_updateState) Task=`699d6680-b4a5-477d-a18a-146b95594b04`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:39,571::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:39,571::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:39,571::task::990::TaskManager.Task::(_decref) Task=`699d6680-b4a5-477d-a18a-146b95594b04`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:40,576::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:40,576::task::595::TaskManager.Task::(_updateState) Task=`fcb57fcb-64b5-48d5-9ca5-73cfef3e9fd0`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:40,577::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:40,577::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:40,577::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:40,577::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:40,578::task::1185::TaskManager.Task::(prepare) Task=`fcb57fcb-64b5-48d5-9ca5-73cfef3e9fd0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:40,578::task::595::TaskManager.Task::(_updateState) Task=`fcb57fcb-64b5-48d5-9ca5-73cfef3e9fd0`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:40,578::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:40,579::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:40,579::task::990::TaskManager.Task::(_decref) Task=`fcb57fcb-64b5-48d5-9ca5-73cfef3e9fd0`::ref 0 aborting False
>Thread-43::DEBUG::2014-04-11 22:57:41,107::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 22:57:41,120::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000465291 s, 1.6 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:57:41,584::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:41,585::task::595::TaskManager.Task::(_updateState) Task=`c7687206-69c4-45f4-a87d-0651121cc0c3`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:41,586::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:41,586::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:41,586::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:41,587::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:41,587::task::1185::TaskManager.Task::(prepare) Task=`c7687206-69c4-45f4-a87d-0651121cc0c3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:41,587::task::595::TaskManager.Task::(_updateState) Task=`c7687206-69c4-45f4-a87d-0651121cc0c3`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:41,588::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:41,588::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:41,588::task::990::TaskManager.Task::(_decref) Task=`c7687206-69c4-45f4-a87d-0651121cc0c3`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:42,594::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:42,595::task::595::TaskManager.Task::(_updateState) Task=`9e1c4762-ace7-4c6b-b88b-5f2f8c29e419`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:42,596::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:42,596::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:42,596::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:42,597::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:42,597::task::1185::TaskManager.Task::(prepare) Task=`9e1c4762-ace7-4c6b-b88b-5f2f8c29e419`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:42,597::task::595::TaskManager.Task::(_updateState) Task=`9e1c4762-ace7-4c6b-b88b-5f2f8c29e419`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:42,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:42,598::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:42,598::task::990::TaskManager.Task::(_decref) Task=`9e1c4762-ace7-4c6b-b88b-5f2f8c29e419`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:57:42,800::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:57:42,813::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000355257 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:57:42,910::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:57:42,924::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000505038 s, 1.4 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:57:43,027::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:57:43,059::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000477959 s, 1.6 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:57:43,091::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:57:43,102::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000370907 s, 2.0 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:57:43,604::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:43,618::task::595::TaskManager.Task::(_updateState) Task=`7cb64084-6b5f-4e85-9321-5aa034fd9b49`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:43,619::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:43,619::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:43,619::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:43,620::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:43,620::task::1185::TaskManager.Task::(prepare) Task=`7cb64084-6b5f-4e85-9321-5aa034fd9b49`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:43,620::task::595::TaskManager.Task::(_updateState) Task=`7cb64084-6b5f-4e85-9321-5aa034fd9b49`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:43,621::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:43,621::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:43,621::task::990::TaskManager.Task::(_decref) Task=`7cb64084-6b5f-4e85-9321-5aa034fd9b49`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:57:43,788::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:57:43,788::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-33::ERROR::2014-04-11 22:57:43,790::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-39::DEBUG::2014-04-11 22:57:43,791::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-33::ERROR::2014-04-11 22:57:43,792::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:57:43,797::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-39::DEBUG::2014-04-11 22:57:43,814::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::DEBUG::2014-04-11 22:57:43,829::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:57:44,091::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:57:44,093::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-33::DEBUG::2014-04-11 22:57:44,129::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-39::ERROR::2014-04-11 22:57:44,132::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:57:44,136::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::WARNING::2014-04-11 22:57:44,141::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:57:44,141::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:57:44,171::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:57:44,172::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:57:44,628::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:44,629::task::595::TaskManager.Task::(_updateState) Task=`d8a17bf8-a475-4e44-b15c-f7d15390d6e1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:44,629::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:44,629::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:44,630::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:44,630::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:44,630::task::1185::TaskManager.Task::(prepare) Task=`d8a17bf8-a475-4e44-b15c-f7d15390d6e1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:44,631::task::595::TaskManager.Task::(_updateState) Task=`d8a17bf8-a475-4e44-b15c-f7d15390d6e1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:44,631::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:44,631::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:44,632::task::990::TaskManager.Task::(_decref) Task=`d8a17bf8-a475-4e44-b15c-f7d15390d6e1`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:45,637::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:45,637::task::595::TaskManager.Task::(_updateState) Task=`d5caadc1-96cc-49f7-b2af-84f4b63d7477`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:45,638::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:45,638::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:45,638::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:45,639::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:45,639::task::1185::TaskManager.Task::(prepare) Task=`d5caadc1-96cc-49f7-b2af-84f4b63d7477`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:45,639::task::595::TaskManager.Task::(_updateState) Task=`d5caadc1-96cc-49f7-b2af-84f4b63d7477`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:45,640::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:45,640::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:45,640::task::990::TaskManager.Task::(_decref) Task=`d5caadc1-96cc-49f7-b2af-84f4b63d7477`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:46,645::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:46,646::task::595::TaskManager.Task::(_updateState) Task=`c155e776-4d7f-4f05-81c5-efbb86246560`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:46,646::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:46,646::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:46,647::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:46,647::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:46,647::task::1185::TaskManager.Task::(prepare) Task=`c155e776-4d7f-4f05-81c5-efbb86246560`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:46,648::task::595::TaskManager.Task::(_updateState) Task=`c155e776-4d7f-4f05-81c5-efbb86246560`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:46,648::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:46,648::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:46,648::task::990::TaskManager.Task::(_decref) Task=`c155e776-4d7f-4f05-81c5-efbb86246560`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:47,654::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:47,655::task::595::TaskManager.Task::(_updateState) Task=`9f5f5b7a-1f80-4574-8528-d82d73b468f9`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:47,656::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:47,656::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:47,656::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:47,657::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:47,657::task::1185::TaskManager.Task::(prepare) Task=`9f5f5b7a-1f80-4574-8528-d82d73b468f9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:47,658::task::595::TaskManager.Task::(_updateState) Task=`9f5f5b7a-1f80-4574-8528-d82d73b468f9`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:47,658::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:47,658::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:47,659::task::990::TaskManager.Task::(_decref) Task=`9f5f5b7a-1f80-4574-8528-d82d73b468f9`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:48,664::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:48,664::task::595::TaskManager.Task::(_updateState) Task=`ffddc5c3-a8cf-4a55-8dcf-7f6cfc08818b`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:48,665::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:48,665::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:48,665::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:48,666::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:48,666::task::1185::TaskManager.Task::(prepare) Task=`ffddc5c3-a8cf-4a55-8dcf-7f6cfc08818b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:48,666::task::595::TaskManager.Task::(_updateState) Task=`ffddc5c3-a8cf-4a55-8dcf-7f6cfc08818b`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:48,667::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:48,667::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:48,667::task::990::TaskManager.Task::(_decref) Task=`ffddc5c3-a8cf-4a55-8dcf-7f6cfc08818b`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:49,672::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:49,672::task::595::TaskManager.Task::(_updateState) Task=`aafa764b-d395-4715-a338-35ed81f036d3`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:49,673::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:49,673::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:49,673::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:49,674::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:49,674::task::1185::TaskManager.Task::(prepare) Task=`aafa764b-d395-4715-a338-35ed81f036d3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:49,674::task::595::TaskManager.Task::(_updateState) Task=`aafa764b-d395-4715-a338-35ed81f036d3`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:49,674::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:49,675::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:49,675::task::990::TaskManager.Task::(_decref) Task=`aafa764b-d395-4715-a338-35ed81f036d3`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:49,905::task::595::TaskManager.Task::(_updateState) Task=`8698fbd6-9158-4654-be8b-40551af3d853`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:49,905::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:57:49,906::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000505038', 'lastCheck': '7.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000355257', 'lastCheck': '7.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000477959', 'lastCheck': '6.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '18.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000370907', 'lastCheck': '6.8', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:57:49,906::task::1185::TaskManager.Task::(prepare) Task=`8698fbd6-9158-4654-be8b-40551af3d853`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000505038', 'lastCheck': '7.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000355257', 'lastCheck': '7.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000477959', 'lastCheck': '6.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '18.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000370907', 'lastCheck': '6.8', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:57:49,907::task::595::TaskManager.Task::(_updateState) Task=`8698fbd6-9158-4654-be8b-40551af3d853`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:49,907::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:49,907::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:49,908::task::990::TaskManager.Task::(_decref) Task=`8698fbd6-9158-4654-be8b-40551af3d853`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:50,679::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:50,680::task::595::TaskManager.Task::(_updateState) Task=`c852c31d-b52c-4430-8218-7fa572ee4709`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:50,680::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:50,681::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:50,681::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:50,681::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:50,681::task::1185::TaskManager.Task::(prepare) Task=`c852c31d-b52c-4430-8218-7fa572ee4709`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:50,682::task::595::TaskManager.Task::(_updateState) Task=`c852c31d-b52c-4430-8218-7fa572ee4709`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:50,682::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:50,682::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:50,683::task::990::TaskManager.Task::(_decref) Task=`c852c31d-b52c-4430-8218-7fa572ee4709`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:51,687::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:51,688::task::595::TaskManager.Task::(_updateState) Task=`732a27a9-45d9-4119-93e5-04723929980a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:51,688::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:51,689::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:51,689::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:51,689::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:51,689::task::1185::TaskManager.Task::(prepare) Task=`732a27a9-45d9-4119-93e5-04723929980a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:51,690::task::595::TaskManager.Task::(_updateState) Task=`732a27a9-45d9-4119-93e5-04723929980a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:51,690::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:51,690::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:51,691::task::990::TaskManager.Task::(_decref) Task=`732a27a9-45d9-4119-93e5-04723929980a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:52,696::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:52,697::task::595::TaskManager.Task::(_updateState) Task=`b95479c8-f1aa-491e-a350-70150a9e56c2`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:52,698::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:52,698::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:52,699::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:52,699::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:52,699::task::1185::TaskManager.Task::(prepare) Task=`b95479c8-f1aa-491e-a350-70150a9e56c2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:52,700::task::595::TaskManager.Task::(_updateState) Task=`b95479c8-f1aa-491e-a350-70150a9e56c2`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:52,700::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:52,700::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:52,701::task::990::TaskManager.Task::(_decref) Task=`b95479c8-f1aa-491e-a350-70150a9e56c2`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:57:52,842::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:57:52,855::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000433634 s, 851 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:57:52,955::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:57:52,968::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000467098 s, 1.5 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:57:53,096::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:57:53,119::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000480641 s, 1.6 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:57:53,153::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:57:53,164::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000400999 s, 1.9 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:57:53,706::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:53,707::task::595::TaskManager.Task::(_updateState) Task=`78686854-5150-444b-9591-d802b97dd454`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:53,708::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:53,708::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:53,708::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:53,709::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:53,709::task::1185::TaskManager.Task::(prepare) Task=`78686854-5150-444b-9591-d802b97dd454`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:53,710::task::595::TaskManager.Task::(_updateState) Task=`78686854-5150-444b-9591-d802b97dd454`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:53,710::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:53,710::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:53,711::task::990::TaskManager.Task::(_decref) Task=`78686854-5150-444b-9591-d802b97dd454`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:57:54,141::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:57:54,141::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:57:54,142::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:57:54,143::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:57:54,173::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:57:54,174::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:57:54,174::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:57:54,176::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:57:54,426::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:57:54,428::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-33::DEBUG::2014-04-11 22:57:54,455::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-39::ERROR::2014-04-11 22:57:54,457::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:57:54,458::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::WARNING::2014-04-11 22:57:54,459::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:57:54,460::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:57:54,468::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:57:54,469::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:57:54,716::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:54,717::task::595::TaskManager.Task::(_updateState) Task=`a710cdcc-d7cd-4b4c-889b-e47c2a89c58d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:54,718::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:54,718::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:54,718::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:54,718::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:54,719::task::1185::TaskManager.Task::(prepare) Task=`a710cdcc-d7cd-4b4c-889b-e47c2a89c58d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:54,719::task::595::TaskManager.Task::(_updateState) Task=`a710cdcc-d7cd-4b4c-889b-e47c2a89c58d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:54,719::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:54,720::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:54,720::task::990::TaskManager.Task::(_decref) Task=`a710cdcc-d7cd-4b4c-889b-e47c2a89c58d`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:55,725::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:55,725::task::595::TaskManager.Task::(_updateState) Task=`4c2b2522-21fe-4e7b-b54b-ef52178052ea`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:55,726::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:55,726::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:55,726::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:55,727::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:55,727::task::1185::TaskManager.Task::(prepare) Task=`4c2b2522-21fe-4e7b-b54b-ef52178052ea`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:55,727::task::595::TaskManager.Task::(_updateState) Task=`4c2b2522-21fe-4e7b-b54b-ef52178052ea`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:55,727::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:55,728::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:55,728::task::990::TaskManager.Task::(_decref) Task=`4c2b2522-21fe-4e7b-b54b-ef52178052ea`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:56,733::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:56,734::task::595::TaskManager.Task::(_updateState) Task=`6e302ab3-c074-48d6-9c55-3bc3c084d8e8`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:56,734::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:56,734::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:56,735::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:56,735::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:56,735::task::1185::TaskManager.Task::(prepare) Task=`6e302ab3-c074-48d6-9c55-3bc3c084d8e8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:56,735::task::595::TaskManager.Task::(_updateState) Task=`6e302ab3-c074-48d6-9c55-3bc3c084d8e8`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:56,736::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:56,736::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:56,736::task::990::TaskManager.Task::(_decref) Task=`6e302ab3-c074-48d6-9c55-3bc3c084d8e8`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:57,742::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:57,743::task::595::TaskManager.Task::(_updateState) Task=`8291e8d2-fc5f-4ffa-ac10-b0f43ba2bbe3`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:57,744::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:57,744::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:57,745::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:57,745::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:57,745::task::1185::TaskManager.Task::(prepare) Task=`8291e8d2-fc5f-4ffa-ac10-b0f43ba2bbe3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:57,746::task::595::TaskManager.Task::(_updateState) Task=`8291e8d2-fc5f-4ffa-ac10-b0f43ba2bbe3`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:57,746::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:57,746::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:57,747::task::990::TaskManager.Task::(_decref) Task=`8291e8d2-fc5f-4ffa-ac10-b0f43ba2bbe3`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:58,752::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:58,753::task::595::TaskManager.Task::(_updateState) Task=`3eee08cc-1b45-4c8d-ba84-b1e69472d38a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:58,753::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:58,753::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:58,754::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:58,754::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:58,754::task::1185::TaskManager.Task::(prepare) Task=`3eee08cc-1b45-4c8d-ba84-b1e69472d38a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:58,755::task::595::TaskManager.Task::(_updateState) Task=`3eee08cc-1b45-4c8d-ba84-b1e69472d38a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:58,755::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:58,755::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:58,756::task::990::TaskManager.Task::(_decref) Task=`3eee08cc-1b45-4c8d-ba84-b1e69472d38a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:57:59,760::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:57:59,761::task::595::TaskManager.Task::(_updateState) Task=`9bb4ad9a-e8f4-4353-98bf-1fe13514d013`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:57:59,762::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:57:59,762::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:57:59,762::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:57:59,763::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:59,763::task::1185::TaskManager.Task::(prepare) Task=`9bb4ad9a-e8f4-4353-98bf-1fe13514d013`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:57:59,763::task::595::TaskManager.Task::(_updateState) Task=`9bb4ad9a-e8f4-4353-98bf-1fe13514d013`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:57:59,764::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:57:59,764::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:57:59,764::task::990::TaskManager.Task::(_decref) Task=`9bb4ad9a-e8f4-4353-98bf-1fe13514d013`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:00,769::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:00,770::task::595::TaskManager.Task::(_updateState) Task=`f7f36f92-1431-46a9-aa56-1fb5bc41665d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:00,770::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:00,770::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:00,771::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:00,771::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:00,771::task::1185::TaskManager.Task::(prepare) Task=`f7f36f92-1431-46a9-aa56-1fb5bc41665d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:00,772::task::595::TaskManager.Task::(_updateState) Task=`f7f36f92-1431-46a9-aa56-1fb5bc41665d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:00,772::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:00,772::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:00,772::task::990::TaskManager.Task::(_decref) Task=`f7f36f92-1431-46a9-aa56-1fb5bc41665d`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:01,778::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:01,778::task::595::TaskManager.Task::(_updateState) Task=`bcb1d20c-2a73-4e9f-bb22-f080c535f984`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:01,779::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:01,779::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:01,779::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:01,780::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:01,780::task::1185::TaskManager.Task::(prepare) Task=`bcb1d20c-2a73-4e9f-bb22-f080c535f984`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:01,780::task::595::TaskManager.Task::(_updateState) Task=`bcb1d20c-2a73-4e9f-bb22-f080c535f984`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:01,781::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:01,781::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:01,781::task::990::TaskManager.Task::(_decref) Task=`bcb1d20c-2a73-4e9f-bb22-f080c535f984`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:02,790::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:02,791::task::595::TaskManager.Task::(_updateState) Task=`3f4bbd25-42d2-46de-9d33-016441c84529`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:02,792::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:02,792::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:02,792::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:02,793::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:02,793::task::1185::TaskManager.Task::(prepare) Task=`3f4bbd25-42d2-46de-9d33-016441c84529`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:02,794::task::595::TaskManager.Task::(_updateState) Task=`3f4bbd25-42d2-46de-9d33-016441c84529`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:02,794::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:02,794::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:02,795::task::990::TaskManager.Task::(_decref) Task=`3f4bbd25-42d2-46de-9d33-016441c84529`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:58:02,883::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:58:02,896::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000409128 s, 902 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:58:02,997::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:58:03,010::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000410639 s, 1.7 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:58:03,156::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:58:03,168::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000456047 s, 1.7 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:58:03,199::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:58:03,211::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000305761 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:03,813::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:03,814::task::595::TaskManager.Task::(_updateState) Task=`80098a57-a998-466a-87fa-2e1f269a54ce`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:03,815::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:03,815::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:03,815::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:03,816::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:03,816::task::1185::TaskManager.Task::(prepare) Task=`80098a57-a998-466a-87fa-2e1f269a54ce`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:03,816::task::595::TaskManager.Task::(_updateState) Task=`80098a57-a998-466a-87fa-2e1f269a54ce`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:03,817::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:03,817::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:03,817::task::990::TaskManager.Task::(_decref) Task=`80098a57-a998-466a-87fa-2e1f269a54ce`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:58:04,459::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:58:04,460::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:58:04,460::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:58:04,461::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:58:04,471::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:58:04,472::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:58:04,473::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:58:04,475::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:58:04,732::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:58:04,734::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-33::DEBUG::2014-04-11 22:58:04,771::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-39::ERROR::2014-04-11 22:58:04,771::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:58:04,776::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::WARNING::2014-04-11 22:58:04,780::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:58:04,781::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:58:04,811::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:58:04,812::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:58:04,823::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:04,824::task::595::TaskManager.Task::(_updateState) Task=`ddb4d96e-0a37-4c15-b31e-3ad06f4536a5`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:04,825::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:04,825::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:04,825::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:04,826::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:04,826::task::1185::TaskManager.Task::(prepare) Task=`ddb4d96e-0a37-4c15-b31e-3ad06f4536a5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:04,827::task::595::TaskManager.Task::(_updateState) Task=`ddb4d96e-0a37-4c15-b31e-3ad06f4536a5`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:04,827::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:04,827::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:04,828::task::990::TaskManager.Task::(_decref) Task=`ddb4d96e-0a37-4c15-b31e-3ad06f4536a5`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:05,235::task::595::TaskManager.Task::(_updateState) Task=`6b29c5e0-c2e2-4dc9-a27a-b65dff86d4b5`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:05,236::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:58:05,237::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000410639', 'lastCheck': '2.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409128', 'lastCheck': '2.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000456047', 'lastCheck': '2.1', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '34.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000305761', 'lastCheck': '2.0', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:58:05,237::task::1185::TaskManager.Task::(prepare) Task=`6b29c5e0-c2e2-4dc9-a27a-b65dff86d4b5`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000410639', 'lastCheck': '2.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409128', 'lastCheck': '2.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000456047', 'lastCheck': '2.1', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '34.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000305761', 'lastCheck': '2.0', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:58:05,238::task::595::TaskManager.Task::(_updateState) Task=`6b29c5e0-c2e2-4dc9-a27a-b65dff86d4b5`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:05,238::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:05,238::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:05,239::task::990::TaskManager.Task::(_decref) Task=`6b29c5e0-c2e2-4dc9-a27a-b65dff86d4b5`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:05,833::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:05,833::task::595::TaskManager.Task::(_updateState) Task=`393f2aa8-2af6-4568-9882-fa49da6d187c`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:05,834::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:05,834::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:05,834::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:05,835::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:05,835::task::1185::TaskManager.Task::(prepare) Task=`393f2aa8-2af6-4568-9882-fa49da6d187c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:05,835::task::595::TaskManager.Task::(_updateState) Task=`393f2aa8-2af6-4568-9882-fa49da6d187c`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:05,836::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:05,836::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:05,836::task::990::TaskManager.Task::(_decref) Task=`393f2aa8-2af6-4568-9882-fa49da6d187c`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:06,841::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:06,842::task::595::TaskManager.Task::(_updateState) Task=`53102cd0-f81d-4f38-ad35-f8ba6d96052a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:06,842::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:06,843::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:06,843::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:06,843::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:06,844::task::1185::TaskManager.Task::(prepare) Task=`53102cd0-f81d-4f38-ad35-f8ba6d96052a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:06,844::task::595::TaskManager.Task::(_updateState) Task=`53102cd0-f81d-4f38-ad35-f8ba6d96052a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:06,844::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:06,844::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:06,845::task::990::TaskManager.Task::(_decref) Task=`53102cd0-f81d-4f38-ad35-f8ba6d96052a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:07,850::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:07,852::task::595::TaskManager.Task::(_updateState) Task=`7283f1d8-b650-4a86-ad29-2de242ee00ca`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:07,852::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:07,853::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:07,853::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:07,853::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:07,854::task::1185::TaskManager.Task::(prepare) Task=`7283f1d8-b650-4a86-ad29-2de242ee00ca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:07,854::task::595::TaskManager.Task::(_updateState) Task=`7283f1d8-b650-4a86-ad29-2de242ee00ca`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:07,855::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:07,855::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:07,855::task::990::TaskManager.Task::(_decref) Task=`7283f1d8-b650-4a86-ad29-2de242ee00ca`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:08,860::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:08,861::task::595::TaskManager.Task::(_updateState) Task=`cd1f4eff-de21-4ce0-a8db-4c319bb3eabf`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:08,862::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:08,862::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:08,862::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:08,863::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:08,863::task::1185::TaskManager.Task::(prepare) Task=`cd1f4eff-de21-4ce0-a8db-4c319bb3eabf`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:08,863::task::595::TaskManager.Task::(_updateState) Task=`cd1f4eff-de21-4ce0-a8db-4c319bb3eabf`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:08,864::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:08,864::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:08,864::task::990::TaskManager.Task::(_decref) Task=`cd1f4eff-de21-4ce0-a8db-4c319bb3eabf`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:09,869::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:09,869::task::595::TaskManager.Task::(_updateState) Task=`c5eca62d-fa5c-4998-8b05-1d8734cfe8cc`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:09,870::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:09,870::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:09,870::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:09,871::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:09,871::task::1185::TaskManager.Task::(prepare) Task=`c5eca62d-fa5c-4998-8b05-1d8734cfe8cc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:09,871::task::595::TaskManager.Task::(_updateState) Task=`c5eca62d-fa5c-4998-8b05-1d8734cfe8cc`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:09,872::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:09,872::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:09,872::task::990::TaskManager.Task::(_decref) Task=`c5eca62d-fa5c-4998-8b05-1d8734cfe8cc`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:10,877::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:10,877::task::595::TaskManager.Task::(_updateState) Task=`f7930e3a-a85e-474f-90d2-2a1f5e14fc73`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:10,878::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:10,878::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:10,878::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:10,879::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:10,879::task::1185::TaskManager.Task::(prepare) Task=`f7930e3a-a85e-474f-90d2-2a1f5e14fc73`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:10,879::task::595::TaskManager.Task::(_updateState) Task=`f7930e3a-a85e-474f-90d2-2a1f5e14fc73`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:10,879::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:10,880::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:10,880::task::990::TaskManager.Task::(_decref) Task=`f7930e3a-a85e-474f-90d2-2a1f5e14fc73`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:11,885::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:11,885::task::595::TaskManager.Task::(_updateState) Task=`63cd38c0-c713-4e74-98fb-f623e1f9ab17`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:11,886::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:11,886::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:11,886::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:11,886::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:11,887::task::1185::TaskManager.Task::(prepare) Task=`63cd38c0-c713-4e74-98fb-f623e1f9ab17`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:11,887::task::595::TaskManager.Task::(_updateState) Task=`63cd38c0-c713-4e74-98fb-f623e1f9ab17`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:11,887::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:11,888::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:11,888::task::990::TaskManager.Task::(_decref) Task=`63cd38c0-c713-4e74-98fb-f623e1f9ab17`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:12,894::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:12,895::task::595::TaskManager.Task::(_updateState) Task=`c14d45b5-9b12-45f5-b375-417a72bcaee1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:12,895::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:12,896::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:12,896::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:12,896::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:12,897::task::1185::TaskManager.Task::(prepare) Task=`c14d45b5-9b12-45f5-b375-417a72bcaee1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:12,897::task::595::TaskManager.Task::(_updateState) Task=`c14d45b5-9b12-45f5-b375-417a72bcaee1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:12,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:12,898::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:12,898::task::990::TaskManager.Task::(_decref) Task=`c14d45b5-9b12-45f5-b375-417a72bcaee1`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:58:12,929::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:58:12,942::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.00038346 s, 962 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:58:13,038::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:58:13,051::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000477924 s, 1.5 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:58:13,200::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:58:13,212::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000459218 s, 1.6 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:58:13,241::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:58:13,252::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000364313 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:13,909::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:13,910::task::595::TaskManager.Task::(_updateState) Task=`970396ac-40c5-4d47-9de9-198c8fbcf2ae`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:13,911::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:13,911::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:13,911::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:13,912::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:13,912::task::1185::TaskManager.Task::(prepare) Task=`970396ac-40c5-4d47-9de9-198c8fbcf2ae`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:13,913::task::595::TaskManager.Task::(_updateState) Task=`970396ac-40c5-4d47-9de9-198c8fbcf2ae`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:13,913::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:13,913::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:13,914::task::990::TaskManager.Task::(_decref) Task=`970396ac-40c5-4d47-9de9-198c8fbcf2ae`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:58:14,781::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:58:14,781::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:58:14,781::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:58:14,783::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:58:14,813::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:58:14,814::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:58:14,815::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:58:14,816::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:58:14,920::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:14,922::task::595::TaskManager.Task::(_updateState) Task=`aa551611-7b34-41fe-9d2a-1089612b6da1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:14,922::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:14,923::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:14,923::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:14,923::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:14,924::task::1185::TaskManager.Task::(prepare) Task=`aa551611-7b34-41fe-9d2a-1089612b6da1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:14,924::task::595::TaskManager.Task::(_updateState) Task=`aa551611-7b34-41fe-9d2a-1089612b6da1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:14,924::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:14,925::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:14,925::task::990::TaskManager.Task::(_decref) Task=`aa551611-7b34-41fe-9d2a-1089612b6da1`::ref 0 aborting False
>Thread-39::DEBUG::2014-04-11 22:58:15,051::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:58:15,053::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:58:15,085::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:58:15,086::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:58:15,088::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:58:15,090::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:58:15,090::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:58:15,119::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:58:15,120::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:58:15,931::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:15,931::task::595::TaskManager.Task::(_updateState) Task=`a8abc817-f46f-4730-a0ee-c7674b438b90`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:15,932::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:15,932::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:15,932::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:15,933::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:15,933::task::1185::TaskManager.Task::(prepare) Task=`a8abc817-f46f-4730-a0ee-c7674b438b90`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:15,933::task::595::TaskManager.Task::(_updateState) Task=`a8abc817-f46f-4730-a0ee-c7674b438b90`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:15,934::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:15,934::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:15,934::task::990::TaskManager.Task::(_decref) Task=`a8abc817-f46f-4730-a0ee-c7674b438b90`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:16,939::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:16,940::task::595::TaskManager.Task::(_updateState) Task=`0499996a-6da6-44d4-b577-022fd8a8a25e`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:16,940::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:16,940::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:16,941::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:16,941::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:16,941::task::1185::TaskManager.Task::(prepare) Task=`0499996a-6da6-44d4-b577-022fd8a8a25e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:16,942::task::595::TaskManager.Task::(_updateState) Task=`0499996a-6da6-44d4-b577-022fd8a8a25e`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:16,942::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:16,943::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:16,943::task::990::TaskManager.Task::(_decref) Task=`0499996a-6da6-44d4-b577-022fd8a8a25e`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:17,948::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:17,950::task::595::TaskManager.Task::(_updateState) Task=`bd45977b-b3f4-497a-89af-a5bbd4ff10cc`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:17,950::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:17,950::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:17,951::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:17,951::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:17,952::task::1185::TaskManager.Task::(prepare) Task=`bd45977b-b3f4-497a-89af-a5bbd4ff10cc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:17,952::task::595::TaskManager.Task::(_updateState) Task=`bd45977b-b3f4-497a-89af-a5bbd4ff10cc`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:17,952::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:17,953::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:17,953::task::990::TaskManager.Task::(_decref) Task=`bd45977b-b3f4-497a-89af-a5bbd4ff10cc`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:18,958::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:18,959::task::595::TaskManager.Task::(_updateState) Task=`0a1fb5d6-7b11-44f3-8d9a-9229e657db5d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:18,959::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:18,959::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:18,960::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:18,960::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:18,960::task::1185::TaskManager.Task::(prepare) Task=`0a1fb5d6-7b11-44f3-8d9a-9229e657db5d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:18,960::task::595::TaskManager.Task::(_updateState) Task=`0a1fb5d6-7b11-44f3-8d9a-9229e657db5d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:18,961::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:18,961::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:18,961::task::990::TaskManager.Task::(_decref) Task=`0a1fb5d6-7b11-44f3-8d9a-9229e657db5d`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:19,966::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:19,982::task::595::TaskManager.Task::(_updateState) Task=`96bc4fb1-ab05-43b7-97dc-8bd112137c51`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:19,983::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:19,983::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:19,983::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:19,984::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:19,984::task::1185::TaskManager.Task::(prepare) Task=`96bc4fb1-ab05-43b7-97dc-8bd112137c51`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:19,985::task::595::TaskManager.Task::(_updateState) Task=`96bc4fb1-ab05-43b7-97dc-8bd112137c51`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:19,985::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:19,985::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:19,986::task::990::TaskManager.Task::(_decref) Task=`96bc4fb1-ab05-43b7-97dc-8bd112137c51`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:20,556::task::595::TaskManager.Task::(_updateState) Task=`4762bfee-b14c-4a98-82be-d191002d4b55`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:20,556::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:58:20,557::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000477924', 'lastCheck': '7.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00038346', 'lastCheck': '7.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000459218', 'lastCheck': '7.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '49.5', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000364313', 'lastCheck': '7.3', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:58:20,557::task::1185::TaskManager.Task::(prepare) Task=`4762bfee-b14c-4a98-82be-d191002d4b55`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000477924', 'lastCheck': '7.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00038346', 'lastCheck': '7.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000459218', 'lastCheck': '7.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '49.5', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000364313', 'lastCheck': '7.3', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:58:20,558::task::595::TaskManager.Task::(_updateState) Task=`4762bfee-b14c-4a98-82be-d191002d4b55`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:20,558::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:20,558::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:20,559::task::990::TaskManager.Task::(_decref) Task=`4762bfee-b14c-4a98-82be-d191002d4b55`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:20,991::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:20,991::task::595::TaskManager.Task::(_updateState) Task=`f4034e85-51d0-427a-8f5e-efd2414f5538`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:20,992::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:20,992::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:20,992::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:20,993::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:20,993::task::1185::TaskManager.Task::(prepare) Task=`f4034e85-51d0-427a-8f5e-efd2414f5538`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:20,993::task::595::TaskManager.Task::(_updateState) Task=`f4034e85-51d0-427a-8f5e-efd2414f5538`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:20,994::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:20,994::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:20,994::task::990::TaskManager.Task::(_decref) Task=`f4034e85-51d0-427a-8f5e-efd2414f5538`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:21,999::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:22,000::task::595::TaskManager.Task::(_updateState) Task=`bfd7f20e-0d30-414f-bff9-13f57c41f89b`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:22,000::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:22,000::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:22,000::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:22,001::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:22,001::task::1185::TaskManager.Task::(prepare) Task=`bfd7f20e-0d30-414f-bff9-13f57c41f89b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:22,001::task::595::TaskManager.Task::(_updateState) Task=`bfd7f20e-0d30-414f-bff9-13f57c41f89b`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:22,002::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:22,002::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:22,002::task::990::TaskManager.Task::(_decref) Task=`bfd7f20e-0d30-414f-bff9-13f57c41f89b`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:58:22,971::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:58:22,989::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000313431 s, 1.2 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:23,008::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:23,009::task::595::TaskManager.Task::(_updateState) Task=`af1907a0-7c03-493d-b0b3-60d9bb633a71`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:23,010::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:23,010::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:23,010::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:23,011::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:23,011::task::1185::TaskManager.Task::(prepare) Task=`af1907a0-7c03-493d-b0b3-60d9bb633a71`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:23,012::task::595::TaskManager.Task::(_updateState) Task=`af1907a0-7c03-493d-b0b3-60d9bb633a71`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:23,012::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:23,012::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:23,013::task::990::TaskManager.Task::(_decref) Task=`af1907a0-7c03-493d-b0b3-60d9bb633a71`::ref 0 aborting False
>Thread-22::DEBUG::2014-04-11 22:58:23,079::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:58:23,093::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000454332 s, 1.6 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:58:23,239::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:58:23,251::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000441898 s, 1.7 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:58:23,282::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:58:23,293::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000285471 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:24,018::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:24,018::task::595::TaskManager.Task::(_updateState) Task=`8c4fee41-f74b-4ec7-bc70-6bb49c552587`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:24,019::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:24,019::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:24,020::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:24,020::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:24,020::task::1185::TaskManager.Task::(prepare) Task=`8c4fee41-f74b-4ec7-bc70-6bb49c552587`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:24,021::task::595::TaskManager.Task::(_updateState) Task=`8c4fee41-f74b-4ec7-bc70-6bb49c552587`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:24,021::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:24,021::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:24,022::task::990::TaskManager.Task::(_decref) Task=`8c4fee41-f74b-4ec7-bc70-6bb49c552587`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:25,026::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:25,046::task::595::TaskManager.Task::(_updateState) Task=`9c683637-cc7c-450b-80c4-ae5818ba9f0d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:25,047::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:25,047::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:25,047::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:25,048::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:25,048::task::1185::TaskManager.Task::(prepare) Task=`9c683637-cc7c-450b-80c4-ae5818ba9f0d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:25,048::task::595::TaskManager.Task::(_updateState) Task=`9c683637-cc7c-450b-80c4-ae5818ba9f0d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:25,048::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:25,049::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:25,049::task::990::TaskManager.Task::(_decref) Task=`9c683637-cc7c-450b-80c4-ae5818ba9f0d`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:58:25,087::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:58:25,087::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:58:25,087::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:58:25,088::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:58:25,121::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:58:25,122::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:58:25,122::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:58:25,123::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:58:25,356::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:58:25,358::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:58:25,392::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:58:25,393::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:58:25,396::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:58:25,398::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:58:25,398::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:58:25,427::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:58:25,428::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:58:26,054::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:26,055::task::595::TaskManager.Task::(_updateState) Task=`0134ab71-3642-4443-8d49-83ea61d0c684`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:26,056::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:26,056::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:26,057::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:26,057::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:26,057::task::1185::TaskManager.Task::(prepare) Task=`0134ab71-3642-4443-8d49-83ea61d0c684`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:26,058::task::595::TaskManager.Task::(_updateState) Task=`0134ab71-3642-4443-8d49-83ea61d0c684`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:26,058::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:26,058::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:26,059::task::990::TaskManager.Task::(_decref) Task=`0134ab71-3642-4443-8d49-83ea61d0c684`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:27,064::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:27,065::task::595::TaskManager.Task::(_updateState) Task=`ab748b99-7754-42cf-94a6-3489a07aa47f`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:27,065::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:27,065::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:27,066::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:27,066::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:27,066::task::1185::TaskManager.Task::(prepare) Task=`ab748b99-7754-42cf-94a6-3489a07aa47f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:27,067::task::595::TaskManager.Task::(_updateState) Task=`ab748b99-7754-42cf-94a6-3489a07aa47f`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:27,067::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:27,067::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:27,068::task::990::TaskManager.Task::(_decref) Task=`ab748b99-7754-42cf-94a6-3489a07aa47f`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:28,073::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:28,074::task::595::TaskManager.Task::(_updateState) Task=`56ca247d-0045-4245-b91c-c721bbbd0310`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:28,075::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:28,075::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:28,076::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:28,076::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:28,076::task::1185::TaskManager.Task::(prepare) Task=`56ca247d-0045-4245-b91c-c721bbbd0310`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:28,077::task::595::TaskManager.Task::(_updateState) Task=`56ca247d-0045-4245-b91c-c721bbbd0310`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:28,077::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:28,078::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:28,078::task::990::TaskManager.Task::(_decref) Task=`56ca247d-0045-4245-b91c-c721bbbd0310`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:29,084::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:29,096::task::595::TaskManager.Task::(_updateState) Task=`4132d974-2466-4e2b-a611-cdd1cc5be908`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:29,096::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:29,097::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:29,097::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:29,097::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:29,098::task::1185::TaskManager.Task::(prepare) Task=`4132d974-2466-4e2b-a611-cdd1cc5be908`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:29,098::task::595::TaskManager.Task::(_updateState) Task=`4132d974-2466-4e2b-a611-cdd1cc5be908`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:29,098::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:29,099::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:29,099::task::990::TaskManager.Task::(_decref) Task=`4132d974-2466-4e2b-a611-cdd1cc5be908`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:30,105::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:30,106::task::595::TaskManager.Task::(_updateState) Task=`bf105be5-eb52-4976-bdd2-005bc9b3c705`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:30,106::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:30,106::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:30,107::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:30,107::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:30,107::task::1185::TaskManager.Task::(prepare) Task=`bf105be5-eb52-4976-bdd2-005bc9b3c705`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:30,108::task::595::TaskManager.Task::(_updateState) Task=`bf105be5-eb52-4976-bdd2-005bc9b3c705`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:30,108::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:30,108::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:30,109::task::990::TaskManager.Task::(_decref) Task=`bf105be5-eb52-4976-bdd2-005bc9b3c705`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:31,114::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:31,115::task::595::TaskManager.Task::(_updateState) Task=`3df757b9-9e35-4ba8-8606-23683df775ee`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:31,115::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:31,115::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:31,116::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:31,116::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:31,116::task::1185::TaskManager.Task::(prepare) Task=`3df757b9-9e35-4ba8-8606-23683df775ee`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:31,117::task::595::TaskManager.Task::(_updateState) Task=`3df757b9-9e35-4ba8-8606-23683df775ee`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:31,117::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:31,117::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:31,117::task::990::TaskManager.Task::(_decref) Task=`3df757b9-9e35-4ba8-8606-23683df775ee`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:32,122::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:32,122::task::595::TaskManager.Task::(_updateState) Task=`3a634006-72fb-4a2f-bc18-ad56ec3f6ff5`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:32,123::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:32,123::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:32,123::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:32,124::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:32,124::task::1185::TaskManager.Task::(prepare) Task=`3a634006-72fb-4a2f-bc18-ad56ec3f6ff5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:32,124::task::595::TaskManager.Task::(_updateState) Task=`3a634006-72fb-4a2f-bc18-ad56ec3f6ff5`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:32,124::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:32,125::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:32,125::task::990::TaskManager.Task::(_decref) Task=`3a634006-72fb-4a2f-bc18-ad56ec3f6ff5`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:58:33,020::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:58:33,033::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000346881 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:58:33,122::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:58:33,137::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:33,142::task::595::TaskManager.Task::(_updateState) Task=`da7044f7-623c-4619-b4d5-6c113aa3f079`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:33,145::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:33,147::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:33,149::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:33,151::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:33,153::task::1185::TaskManager.Task::(prepare) Task=`da7044f7-623c-4619-b4d5-6c113aa3f079`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:33,156::task::595::TaskManager.Task::(_updateState) Task=`da7044f7-623c-4619-b4d5-6c113aa3f079`::moving from state preparing -> state finished
>Thread-22::DEBUG::2014-04-11 22:58:33,157::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000510101 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:33,158::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:33,161::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:33,162::task::990::TaskManager.Task::(_decref) Task=`da7044f7-623c-4619-b4d5-6c113aa3f079`::ref 0 aborting False
>Thread-29::DEBUG::2014-04-11 22:58:33,279::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:58:33,291::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000472078 s, 1.6 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:58:33,323::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:58:33,335::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000320522 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:34,172::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:34,173::task::595::TaskManager.Task::(_updateState) Task=`cc69ed9d-d654-4c11-a693-d3a5a6cc1af8`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:34,173::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:34,174::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:34,174::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:34,174::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:34,174::task::1185::TaskManager.Task::(prepare) Task=`cc69ed9d-d654-4c11-a693-d3a5a6cc1af8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:34,175::task::595::TaskManager.Task::(_updateState) Task=`cc69ed9d-d654-4c11-a693-d3a5a6cc1af8`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:34,175::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:34,175::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:34,176::task::990::TaskManager.Task::(_decref) Task=`cc69ed9d-d654-4c11-a693-d3a5a6cc1af8`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:35,180::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:35,181::task::595::TaskManager.Task::(_updateState) Task=`2edb7202-8919-475d-8b48-df18f244aef0`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:35,181::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:35,181::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:35,182::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:35,182::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:35,182::task::1185::TaskManager.Task::(prepare) Task=`2edb7202-8919-475d-8b48-df18f244aef0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:35,183::task::595::TaskManager.Task::(_updateState) Task=`2edb7202-8919-475d-8b48-df18f244aef0`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:35,183::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:35,183::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:35,183::task::990::TaskManager.Task::(_decref) Task=`2edb7202-8919-475d-8b48-df18f244aef0`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:58:35,394::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:58:35,394::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:58:35,395::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:58:35,395::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:58:35,429::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:58:35,430::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:58:35,430::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:58:35,431::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:58:35,666::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:58:35,668::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:58:35,704::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:58:35,706::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:58:35,712::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:58:35,714::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:58:35,714::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:58:35,745::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:58:35,746::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:58:35,879::task::595::TaskManager.Task::(_updateState) Task=`db6372b5-758e-4df7-83da-b53d8b80cd74`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:35,879::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:58:35,880::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000510101', 'lastCheck': '2.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346881', 'lastCheck': '2.8', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000472078', 'lastCheck': '2.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '64.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000320522', 'lastCheck': '2.5', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:58:35,881::task::1185::TaskManager.Task::(prepare) Task=`db6372b5-758e-4df7-83da-b53d8b80cd74`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000510101', 'lastCheck': '2.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346881', 'lastCheck': '2.8', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000472078', 'lastCheck': '2.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '64.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000320522', 'lastCheck': '2.5', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:58:35,881::task::595::TaskManager.Task::(_updateState) Task=`db6372b5-758e-4df7-83da-b53d8b80cd74`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:35,881::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:35,882::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:35,882::task::990::TaskManager.Task::(_decref) Task=`db6372b5-758e-4df7-83da-b53d8b80cd74`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:36,188::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:36,189::task::595::TaskManager.Task::(_updateState) Task=`c7158d77-ee39-4bd7-89f7-96c7d5e201b2`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:36,189::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:36,189::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:36,190::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:36,190::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:36,190::task::1185::TaskManager.Task::(prepare) Task=`c7158d77-ee39-4bd7-89f7-96c7d5e201b2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:36,191::task::595::TaskManager.Task::(_updateState) Task=`c7158d77-ee39-4bd7-89f7-96c7d5e201b2`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:36,191::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:36,191::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:36,192::task::990::TaskManager.Task::(_decref) Task=`c7158d77-ee39-4bd7-89f7-96c7d5e201b2`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:37,196::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:37,197::task::595::TaskManager.Task::(_updateState) Task=`3e5da922-0b68-4748-829b-a1cd5e7c1ae2`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:37,197::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:37,198::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:37,198::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:37,198::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:37,198::task::1185::TaskManager.Task::(prepare) Task=`3e5da922-0b68-4748-829b-a1cd5e7c1ae2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:37,199::task::595::TaskManager.Task::(_updateState) Task=`3e5da922-0b68-4748-829b-a1cd5e7c1ae2`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:37,199::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:37,199::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:37,200::task::990::TaskManager.Task::(_decref) Task=`3e5da922-0b68-4748-829b-a1cd5e7c1ae2`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:38,205::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:38,206::task::595::TaskManager.Task::(_updateState) Task=`6f829d28-f7b2-429c-9b06-79518353d352`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:38,207::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:38,207::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:38,208::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:38,208::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:38,208::task::1185::TaskManager.Task::(prepare) Task=`6f829d28-f7b2-429c-9b06-79518353d352`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:38,209::task::595::TaskManager.Task::(_updateState) Task=`6f829d28-f7b2-429c-9b06-79518353d352`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:38,209::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:38,209::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:38,210::task::990::TaskManager.Task::(_decref) Task=`6f829d28-f7b2-429c-9b06-79518353d352`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:39,215::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:39,216::task::595::TaskManager.Task::(_updateState) Task=`ba2f3462-a98a-401a-b379-a33a6923ff49`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:39,216::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:39,216::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:39,217::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:39,217::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:39,217::task::1185::TaskManager.Task::(prepare) Task=`ba2f3462-a98a-401a-b379-a33a6923ff49`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:39,218::task::595::TaskManager.Task::(_updateState) Task=`ba2f3462-a98a-401a-b379-a33a6923ff49`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:39,218::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:39,218::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:39,219::task::990::TaskManager.Task::(_decref) Task=`ba2f3462-a98a-401a-b379-a33a6923ff49`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:40,223::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:40,224::task::595::TaskManager.Task::(_updateState) Task=`323d7ae6-6411-48ac-99cb-2e71a04a975a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:40,224::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:40,224::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:40,225::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:40,225::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:40,225::task::1185::TaskManager.Task::(prepare) Task=`323d7ae6-6411-48ac-99cb-2e71a04a975a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:40,226::task::595::TaskManager.Task::(_updateState) Task=`323d7ae6-6411-48ac-99cb-2e71a04a975a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:40,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:40,226::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:40,226::task::990::TaskManager.Task::(_decref) Task=`323d7ae6-6411-48ac-99cb-2e71a04a975a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:41,231::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:41,232::task::595::TaskManager.Task::(_updateState) Task=`4200bb4c-1553-4cee-bd93-3f409033c609`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:41,233::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:41,233::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:41,233::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:41,233::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:41,234::task::1185::TaskManager.Task::(prepare) Task=`4200bb4c-1553-4cee-bd93-3f409033c609`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:41,234::task::595::TaskManager.Task::(_updateState) Task=`4200bb4c-1553-4cee-bd93-3f409033c609`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:41,234::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:41,235::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:41,235::task::990::TaskManager.Task::(_decref) Task=`4200bb4c-1553-4cee-bd93-3f409033c609`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:42,239::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:42,240::task::595::TaskManager.Task::(_updateState) Task=`f5d45540-bb27-4559-8d30-08259c974650`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:42,241::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:42,241::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:42,241::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:42,241::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:42,242::task::1185::TaskManager.Task::(prepare) Task=`f5d45540-bb27-4559-8d30-08259c974650`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:42,242::task::595::TaskManager.Task::(_updateState) Task=`f5d45540-bb27-4559-8d30-08259c974650`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:42,242::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:42,243::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:42,243::task::990::TaskManager.Task::(_decref) Task=`f5d45540-bb27-4559-8d30-08259c974650`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:58:43,062::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:58:43,075::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000346237 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:58:43,193::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:58:43,206::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.00049129 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:43,248::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:43,249::task::595::TaskManager.Task::(_updateState) Task=`23d62849-b7bc-4c74-8279-49a3e9e28eda`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:43,250::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:43,250::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:43,250::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:43,251::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:43,251::task::1185::TaskManager.Task::(prepare) Task=`23d62849-b7bc-4c74-8279-49a3e9e28eda`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:43,251::task::595::TaskManager.Task::(_updateState) Task=`23d62849-b7bc-4c74-8279-49a3e9e28eda`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:43,252::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:43,252::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:43,252::task::990::TaskManager.Task::(_decref) Task=`23d62849-b7bc-4c74-8279-49a3e9e28eda`::ref 0 aborting False
>Thread-29::DEBUG::2014-04-11 22:58:43,319::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:58:43,331::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000465492 s, 1.6 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:58:43,364::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:58:43,376::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000346317 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:44,258::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:44,259::task::595::TaskManager.Task::(_updateState) Task=`4cd0c327-5a64-473d-b8d1-4b32c111f17c`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:44,259::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:44,260::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:44,260::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:44,260::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:44,261::task::1185::TaskManager.Task::(prepare) Task=`4cd0c327-5a64-473d-b8d1-4b32c111f17c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:44,261::task::595::TaskManager.Task::(_updateState) Task=`4cd0c327-5a64-473d-b8d1-4b32c111f17c`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:44,261::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:44,262::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:44,262::task::990::TaskManager.Task::(_decref) Task=`4cd0c327-5a64-473d-b8d1-4b32c111f17c`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:45,267::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:45,267::task::595::TaskManager.Task::(_updateState) Task=`105d6394-83db-445f-be58-2f47e5353350`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:45,268::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:45,268::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:45,269::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:45,269::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:45,269::task::1185::TaskManager.Task::(prepare) Task=`105d6394-83db-445f-be58-2f47e5353350`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:45,270::task::595::TaskManager.Task::(_updateState) Task=`105d6394-83db-445f-be58-2f47e5353350`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:45,270::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:45,270::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:45,271::task::990::TaskManager.Task::(_decref) Task=`105d6394-83db-445f-be58-2f47e5353350`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:58:45,712::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:58:45,712::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:58:45,713::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:58:45,714::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:58:45,747::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:58:45,748::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:58:45,748::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:58:45,749::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:58:45,981::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:58:45,983::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:58:46,021::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:58:46,022::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:58:46,046::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:58:46,048::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:58:46,049::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:58:46,077::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:58:46,078::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:58:46,276::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:46,277::task::595::TaskManager.Task::(_updateState) Task=`ef181863-c764-4b28-9bb7-cbdd69d30a9c`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:46,278::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:46,278::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:46,278::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:46,279::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:46,279::task::1185::TaskManager.Task::(prepare) Task=`ef181863-c764-4b28-9bb7-cbdd69d30a9c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:46,279::task::595::TaskManager.Task::(_updateState) Task=`ef181863-c764-4b28-9bb7-cbdd69d30a9c`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:46,280::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:46,280::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:46,280::task::990::TaskManager.Task::(_decref) Task=`ef181863-c764-4b28-9bb7-cbdd69d30a9c`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:47,285::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:47,286::task::595::TaskManager.Task::(_updateState) Task=`d217fba9-3564-4248-b266-c2eb7bd74cc1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:47,287::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:47,287::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:47,287::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:47,288::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:47,288::task::1185::TaskManager.Task::(prepare) Task=`d217fba9-3564-4248-b266-c2eb7bd74cc1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:47,288::task::595::TaskManager.Task::(_updateState) Task=`d217fba9-3564-4248-b266-c2eb7bd74cc1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:47,289::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:47,289::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:47,289::task::990::TaskManager.Task::(_decref) Task=`d217fba9-3564-4248-b266-c2eb7bd74cc1`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:48,294::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:48,295::task::595::TaskManager.Task::(_updateState) Task=`fe290b89-9977-4d14-81b7-03d66a490d58`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:48,296::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:48,296::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:48,296::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:48,297::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:48,297::task::1185::TaskManager.Task::(prepare) Task=`fe290b89-9977-4d14-81b7-03d66a490d58`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:48,297::task::595::TaskManager.Task::(_updateState) Task=`fe290b89-9977-4d14-81b7-03d66a490d58`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:48,298::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:48,298::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:48,298::task::990::TaskManager.Task::(_decref) Task=`fe290b89-9977-4d14-81b7-03d66a490d58`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:49,303::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:49,304::task::595::TaskManager.Task::(_updateState) Task=`5f40abae-cea1-4b1d-9c48-d34f6f0933ec`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:49,304::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:49,304::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:49,304::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:49,305::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:49,305::task::1185::TaskManager.Task::(prepare) Task=`5f40abae-cea1-4b1d-9c48-d34f6f0933ec`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:49,305::task::595::TaskManager.Task::(_updateState) Task=`5f40abae-cea1-4b1d-9c48-d34f6f0933ec`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:49,306::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:49,306::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:49,306::task::990::TaskManager.Task::(_decref) Task=`5f40abae-cea1-4b1d-9c48-d34f6f0933ec`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:50,311::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:50,311::task::595::TaskManager.Task::(_updateState) Task=`9fe8e2c0-fb33-4eb2-b154-9f677faa21d0`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:50,312::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:50,312::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:50,312::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:50,312::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:50,313::task::1185::TaskManager.Task::(prepare) Task=`9fe8e2c0-fb33-4eb2-b154-9f677faa21d0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:50,313::task::595::TaskManager.Task::(_updateState) Task=`9fe8e2c0-fb33-4eb2-b154-9f677faa21d0`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:50,313::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:50,314::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:50,314::task::990::TaskManager.Task::(_decref) Task=`9fe8e2c0-fb33-4eb2-b154-9f677faa21d0`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:51,183::task::595::TaskManager.Task::(_updateState) Task=`5caa8434-eb80-4718-ba19-f75004553fb6`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:51,183::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:58:51,184::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00049129', 'lastCheck': '8.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346237', 'lastCheck': '8.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000465492', 'lastCheck': '7.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '80.1', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000346317', 'lastCheck': '7.8', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:58:51,184::task::1185::TaskManager.Task::(prepare) Task=`5caa8434-eb80-4718-ba19-f75004553fb6`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00049129', 'lastCheck': '8.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346237', 'lastCheck': '8.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000465492', 'lastCheck': '7.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '80.1', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000346317', 'lastCheck': '7.8', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:58:51,185::task::595::TaskManager.Task::(_updateState) Task=`5caa8434-eb80-4718-ba19-f75004553fb6`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:51,185::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:51,185::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:51,186::task::990::TaskManager.Task::(_decref) Task=`5caa8434-eb80-4718-ba19-f75004553fb6`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:51,318::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:51,319::task::595::TaskManager.Task::(_updateState) Task=`3341692a-6936-429c-a8f9-ea476b959f96`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:51,319::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:51,319::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:51,320::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:51,320::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:51,320::task::1185::TaskManager.Task::(prepare) Task=`3341692a-6936-429c-a8f9-ea476b959f96`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:51,321::task::595::TaskManager.Task::(_updateState) Task=`3341692a-6936-429c-a8f9-ea476b959f96`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:51,321::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:51,321::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:51,321::task::990::TaskManager.Task::(_decref) Task=`3341692a-6936-429c-a8f9-ea476b959f96`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:52,326::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:52,326::task::595::TaskManager.Task::(_updateState) Task=`f14785b4-9af1-4d4e-b104-21e567cc7e6b`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:52,327::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:52,327::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:52,327::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:52,328::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:52,328::task::1185::TaskManager.Task::(prepare) Task=`f14785b4-9af1-4d4e-b104-21e567cc7e6b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:52,328::task::595::TaskManager.Task::(_updateState) Task=`f14785b4-9af1-4d4e-b104-21e567cc7e6b`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:52,328::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:52,329::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:52,329::task::990::TaskManager.Task::(_decref) Task=`f14785b4-9af1-4d4e-b104-21e567cc7e6b`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:58:53,103::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:58:53,116::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000416235 s, 887 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:58:53,234::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:58:53,248::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.0004171 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:53,337::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:53,343::task::595::TaskManager.Task::(_updateState) Task=`5ca7d2ea-0874-4d52-88bf-f7920bd0e1b6`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:53,348::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:53,351::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:53,354::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:53,358::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:53,361::task::1185::TaskManager.Task::(prepare) Task=`5ca7d2ea-0874-4d52-88bf-f7920bd0e1b6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:53,365::task::595::TaskManager.Task::(_updateState) Task=`5ca7d2ea-0874-4d52-88bf-f7920bd0e1b6`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:53,369::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:53,372::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:53,376::task::990::TaskManager.Task::(_decref) Task=`5ca7d2ea-0874-4d52-88bf-f7920bd0e1b6`::ref 0 aborting False
>Thread-29::DEBUG::2014-04-11 22:58:53,549::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:58:53,583::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000389616 s, 1.9 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:58:53,595::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:58:53,607::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000351646 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:58:54,396::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:54,397::task::595::TaskManager.Task::(_updateState) Task=`8ebd4e2a-aa9f-45ee-b0ee-655df4c55b10`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:54,397::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:54,398::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:54,398::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:54,398::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:54,399::task::1185::TaskManager.Task::(prepare) Task=`8ebd4e2a-aa9f-45ee-b0ee-655df4c55b10`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:54,399::task::595::TaskManager.Task::(_updateState) Task=`8ebd4e2a-aa9f-45ee-b0ee-655df4c55b10`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:54,399::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:54,400::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:54,400::task::990::TaskManager.Task::(_decref) Task=`8ebd4e2a-aa9f-45ee-b0ee-655df4c55b10`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:55,405::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:55,406::task::595::TaskManager.Task::(_updateState) Task=`b52bd59a-ddfd-4c49-a813-b764ed9d2b9f`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:55,406::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:55,406::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:55,406::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:55,407::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:55,407::task::1185::TaskManager.Task::(prepare) Task=`b52bd59a-ddfd-4c49-a813-b764ed9d2b9f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:55,407::task::595::TaskManager.Task::(_updateState) Task=`b52bd59a-ddfd-4c49-a813-b764ed9d2b9f`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:55,408::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:55,408::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:55,408::task::990::TaskManager.Task::(_decref) Task=`b52bd59a-ddfd-4c49-a813-b764ed9d2b9f`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:58:56,023::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:58:56,023::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:58:56,023::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:58:56,024::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:58:56,079::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:58:56,080::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:58:56,081::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:58:56,081::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:58:56,293::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:58:56,296::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:58:56,332::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:58:56,333::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:58:56,354::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:58:56,356::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:58:56,356::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:58:56,386::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:58:56,387::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:58:56,414::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:56,415::task::595::TaskManager.Task::(_updateState) Task=`cef5d27f-2dbc-40e1-ba71-e77cd3dbe085`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:56,415::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:56,416::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:56,416::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:56,416::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:56,417::task::1185::TaskManager.Task::(prepare) Task=`cef5d27f-2dbc-40e1-ba71-e77cd3dbe085`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:56,417::task::595::TaskManager.Task::(_updateState) Task=`cef5d27f-2dbc-40e1-ba71-e77cd3dbe085`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:56,417::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:56,418::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:56,418::task::990::TaskManager.Task::(_decref) Task=`cef5d27f-2dbc-40e1-ba71-e77cd3dbe085`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:57,423::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:57,424::task::595::TaskManager.Task::(_updateState) Task=`d56abd21-503a-4ecc-82f4-8240030dcbc2`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:57,424::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:57,425::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:57,425::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:57,425::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:57,426::task::1185::TaskManager.Task::(prepare) Task=`d56abd21-503a-4ecc-82f4-8240030dcbc2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:57,426::task::595::TaskManager.Task::(_updateState) Task=`d56abd21-503a-4ecc-82f4-8240030dcbc2`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:57,426::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:57,427::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:57,427::task::990::TaskManager.Task::(_decref) Task=`d56abd21-503a-4ecc-82f4-8240030dcbc2`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:58,433::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:58,434::task::595::TaskManager.Task::(_updateState) Task=`daff0f4a-b3c4-45a0-8f76-ea5f272297fb`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:58,434::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:58,435::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:58,435::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:58,435::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:58,436::task::1185::TaskManager.Task::(prepare) Task=`daff0f4a-b3c4-45a0-8f76-ea5f272297fb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:58,436::task::595::TaskManager.Task::(_updateState) Task=`daff0f4a-b3c4-45a0-8f76-ea5f272297fb`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:58,436::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:58,437::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:58,437::task::990::TaskManager.Task::(_decref) Task=`daff0f4a-b3c4-45a0-8f76-ea5f272297fb`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:58:59,442::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:58:59,443::task::595::TaskManager.Task::(_updateState) Task=`8f3a36a8-628d-4dad-bf78-b70bf8ade1b3`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:58:59,443::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:58:59,444::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:58:59,444::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:58:59,444::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:59,445::task::1185::TaskManager.Task::(prepare) Task=`8f3a36a8-628d-4dad-bf78-b70bf8ade1b3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:58:59,445::task::595::TaskManager.Task::(_updateState) Task=`8f3a36a8-628d-4dad-bf78-b70bf8ade1b3`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:58:59,445::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:58:59,446::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:58:59,446::task::990::TaskManager.Task::(_decref) Task=`8f3a36a8-628d-4dad-bf78-b70bf8ade1b3`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:00,450::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:00,451::task::595::TaskManager.Task::(_updateState) Task=`3eb90c3d-3702-4b1b-8d5f-5a52c3f2af2e`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:00,452::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:00,452::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:00,452::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:00,453::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:00,453::task::1185::TaskManager.Task::(prepare) Task=`3eb90c3d-3702-4b1b-8d5f-5a52c3f2af2e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:00,453::task::595::TaskManager.Task::(_updateState) Task=`3eb90c3d-3702-4b1b-8d5f-5a52c3f2af2e`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:00,454::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:00,454::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:00,454::task::990::TaskManager.Task::(_decref) Task=`3eb90c3d-3702-4b1b-8d5f-5a52c3f2af2e`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:01,498::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:01,498::task::595::TaskManager.Task::(_updateState) Task=`55443542-eb2d-4d8c-af17-c36aed10282a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:01,499::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:01,499::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:01,499::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:01,500::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:01,500::task::1185::TaskManager.Task::(prepare) Task=`55443542-eb2d-4d8c-af17-c36aed10282a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:01,500::task::595::TaskManager.Task::(_updateState) Task=`55443542-eb2d-4d8c-af17-c36aed10282a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:01,501::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:01,501::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:01,501::task::990::TaskManager.Task::(_decref) Task=`55443542-eb2d-4d8c-af17-c36aed10282a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:02,506::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:02,507::task::595::TaskManager.Task::(_updateState) Task=`e7b077fd-1e5f-4159-863b-9bc66e8aa666`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:02,507::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:02,508::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:02,508::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:02,508::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:02,509::task::1185::TaskManager.Task::(prepare) Task=`e7b077fd-1e5f-4159-863b-9bc66e8aa666`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:02,509::task::595::TaskManager.Task::(_updateState) Task=`e7b077fd-1e5f-4159-863b-9bc66e8aa666`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:02,509::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:02,509::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:02,510::task::990::TaskManager.Task::(_decref) Task=`e7b077fd-1e5f-4159-863b-9bc66e8aa666`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:59:03,141::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:59:03,165::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000312911 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:59:03,276::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:59:03,288::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000422252 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:59:03,515::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:03,516::task::595::TaskManager.Task::(_updateState) Task=`d02b3a6e-a49d-44da-8601-950ecffc5aae`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:03,516::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:03,516::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:03,517::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:03,517::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:03,517::task::1185::TaskManager.Task::(prepare) Task=`d02b3a6e-a49d-44da-8601-950ecffc5aae`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:03,518::task::595::TaskManager.Task::(_updateState) Task=`d02b3a6e-a49d-44da-8601-950ecffc5aae`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:03,518::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:03,518::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:03,519::task::990::TaskManager.Task::(_decref) Task=`d02b3a6e-a49d-44da-8601-950ecffc5aae`::ref 0 aborting False
>Thread-29::DEBUG::2014-04-11 22:59:03,618::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:59:03,652::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000466251 s, 1.6 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:59:03,679::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:59:03,692::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00034126 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:59:04,524::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:04,525::task::595::TaskManager.Task::(_updateState) Task=`465ac12e-8e89-439d-8c4c-5e55aa2c9875`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:04,526::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:04,526::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:04,527::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:04,527::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:04,527::task::1185::TaskManager.Task::(prepare) Task=`465ac12e-8e89-439d-8c4c-5e55aa2c9875`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:04,528::task::595::TaskManager.Task::(_updateState) Task=`465ac12e-8e89-439d-8c4c-5e55aa2c9875`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:04,528::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:04,528::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:04,529::task::990::TaskManager.Task::(_decref) Task=`465ac12e-8e89-439d-8c4c-5e55aa2c9875`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:05,534::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:05,535::task::595::TaskManager.Task::(_updateState) Task=`ca104189-6abd-41ad-8613-57ea0a6e97e7`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:05,535::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:05,535::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:05,536::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:05,536::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:05,536::task::1185::TaskManager.Task::(prepare) Task=`ca104189-6abd-41ad-8613-57ea0a6e97e7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:05,537::task::595::TaskManager.Task::(_updateState) Task=`ca104189-6abd-41ad-8613-57ea0a6e97e7`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:05,537::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:05,537::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:05,538::task::990::TaskManager.Task::(_decref) Task=`ca104189-6abd-41ad-8613-57ea0a6e97e7`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:59:06,334::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:59:06,334::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:59:06,347::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:59:06,348::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:59:06,388::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:59:06,389::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:59:06,389::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:59:06,390::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:59:06,513::task::595::TaskManager.Task::(_updateState) Task=`8148c05e-7eef-4e48-affc-40f8413e6ab1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:06,514::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:59:06,517::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000422252', 'lastCheck': '3.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000312911', 'lastCheck': '3.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000466251', 'lastCheck': '2.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '95.4', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00034126', 'lastCheck': '2.8', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:59:06,517::task::1185::TaskManager.Task::(prepare) Task=`8148c05e-7eef-4e48-affc-40f8413e6ab1`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000422252', 'lastCheck': '3.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000312911', 'lastCheck': '3.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000466251', 'lastCheck': '2.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '95.4', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00034126', 'lastCheck': '2.8', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:59:06,521::task::595::TaskManager.Task::(_updateState) Task=`8148c05e-7eef-4e48-affc-40f8413e6ab1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:06,525::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:06,527::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:06,528::task::990::TaskManager.Task::(_decref) Task=`8148c05e-7eef-4e48-affc-40f8413e6ab1`::ref 0 aborting False
>Thread-15::DEBUG::2014-04-11 22:59:06,543::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-15::DEBUG::2014-04-11 22:59:06,544::task::595::TaskManager.Task::(_updateState) Task=`f664a9a1-c956-4f67-aaaf-aa4280570da5`::moving from state init -> state preparing
>Thread-15::INFO::2014-04-11 22:59:06,545::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-15::DEBUG::2014-04-11 22:59:06,565::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-15::DEBUG::2014-04-11 22:59:06,586::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-15::INFO::2014-04-11 22:59:06,593::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-15::DEBUG::2014-04-11 22:59:06,593::task::1185::TaskManager.Task::(prepare) Task=`f664a9a1-c956-4f67-aaaf-aa4280570da5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-15::DEBUG::2014-04-11 22:59:06,594::task::595::TaskManager.Task::(_updateState) Task=`f664a9a1-c956-4f67-aaaf-aa4280570da5`::moving from state preparing -> state finished
>Thread-15::DEBUG::2014-04-11 22:59:06,594::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15::DEBUG::2014-04-11 22:59:06,595::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15::DEBUG::2014-04-11 22:59:06,596::task::990::TaskManager.Task::(_decref) Task=`f664a9a1-c956-4f67-aaaf-aa4280570da5`::ref 0 aborting False
>Thread-39::DEBUG::2014-04-11 22:59:06,641::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:59:06,643::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:59:06,685::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:59:06,688::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:59:06,689::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:59:06,693::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:59:06,693::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:59:06,722::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:59:06,723::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:59:07,603::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:07,603::task::595::TaskManager.Task::(_updateState) Task=`6a4e9f54-fb9e-4d66-8caf-8db62df964ce`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:07,604::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:07,604::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:07,605::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:07,605::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:07,605::task::1185::TaskManager.Task::(prepare) Task=`6a4e9f54-fb9e-4d66-8caf-8db62df964ce`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:07,606::task::595::TaskManager.Task::(_updateState) Task=`6a4e9f54-fb9e-4d66-8caf-8db62df964ce`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:07,606::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:07,606::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:07,607::task::990::TaskManager.Task::(_decref) Task=`6a4e9f54-fb9e-4d66-8caf-8db62df964ce`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:08,615::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:08,619::task::595::TaskManager.Task::(_updateState) Task=`26c05cfc-757a-4539-a5d2-017b512322d7`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:08,622::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:08,623::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:08,625::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:08,626::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:08,628::task::1185::TaskManager.Task::(prepare) Task=`26c05cfc-757a-4539-a5d2-017b512322d7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:08,630::task::595::TaskManager.Task::(_updateState) Task=`26c05cfc-757a-4539-a5d2-017b512322d7`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:08,631::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:08,631::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:08,634::task::990::TaskManager.Task::(_decref) Task=`26c05cfc-757a-4539-a5d2-017b512322d7`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:09,639::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:09,640::task::595::TaskManager.Task::(_updateState) Task=`da7ff334-b63c-4b9a-aa42-c35d3662bee3`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:09,640::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:09,640::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:09,640::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:09,641::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:09,641::task::1185::TaskManager.Task::(prepare) Task=`da7ff334-b63c-4b9a-aa42-c35d3662bee3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:09,641::task::595::TaskManager.Task::(_updateState) Task=`da7ff334-b63c-4b9a-aa42-c35d3662bee3`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:09,642::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:09,642::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:09,642::task::990::TaskManager.Task::(_decref) Task=`da7ff334-b63c-4b9a-aa42-c35d3662bee3`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:10,647::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:10,647::task::595::TaskManager.Task::(_updateState) Task=`f75bbfa4-db24-41c4-a9af-745793adb20c`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:10,648::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:10,648::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:10,648::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:10,648::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:10,649::task::1185::TaskManager.Task::(prepare) Task=`f75bbfa4-db24-41c4-a9af-745793adb20c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:10,649::task::595::TaskManager.Task::(_updateState) Task=`f75bbfa4-db24-41c4-a9af-745793adb20c`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:10,649::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:10,650::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:10,650::task::990::TaskManager.Task::(_decref) Task=`f75bbfa4-db24-41c4-a9af-745793adb20c`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:11,654::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:11,693::task::595::TaskManager.Task::(_updateState) Task=`9096f25e-d876-4467-bb09-3e1753b75059`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:11,694::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:11,694::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:11,694::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:11,695::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:11,695::task::1185::TaskManager.Task::(prepare) Task=`9096f25e-d876-4467-bb09-3e1753b75059`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:11,695::task::595::TaskManager.Task::(_updateState) Task=`9096f25e-d876-4467-bb09-3e1753b75059`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:11,696::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:11,696::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:11,696::task::990::TaskManager.Task::(_decref) Task=`9096f25e-d876-4467-bb09-3e1753b75059`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:12,701::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:12,701::task::595::TaskManager.Task::(_updateState) Task=`e2df0a0e-ff8e-42aa-9812-cdb30b19fe62`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:12,701::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:12,702::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:12,702::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:12,702::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:12,703::task::1185::TaskManager.Task::(prepare) Task=`e2df0a0e-ff8e-42aa-9812-cdb30b19fe62`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:12,703::task::595::TaskManager.Task::(_updateState) Task=`e2df0a0e-ff8e-42aa-9812-cdb30b19fe62`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:12,703::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:12,703::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:12,704::task::990::TaskManager.Task::(_decref) Task=`e2df0a0e-ff8e-42aa-9812-cdb30b19fe62`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:59:13,190::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:59:13,203::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000413842 s, 892 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:59:13,323::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:59:13,335::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.00048539 s, 1.5 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:59:13,686::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:59:13,715::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:13,722::task::595::TaskManager.Task::(_updateState) Task=`9bd60b08-8f44-44bb-a4e6-2aa554879e49`::moving from state init -> state preparing
>Thread-29::DEBUG::2014-04-11 22:59:13,726::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000552306 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14::INFO::2014-04-11 22:59:13,727::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:13,732::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:13,736::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:13,740::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:13,745::task::1185::TaskManager.Task::(prepare) Task=`9bd60b08-8f44-44bb-a4e6-2aa554879e49`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:13,747::task::595::TaskManager.Task::(_updateState) Task=`9bd60b08-8f44-44bb-a4e6-2aa554879e49`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:13,750::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:13,752::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:13,755::task::990::TaskManager.Task::(_decref) Task=`9bd60b08-8f44-44bb-a4e6-2aa554879e49`::ref 0 aborting False
>Thread-46::DEBUG::2014-04-11 22:59:13,782::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:59:13,794::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000348219 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:59:14,767::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:14,768::task::595::TaskManager.Task::(_updateState) Task=`14e1e225-744d-41b8-a1d7-d6a25aa4a4d4`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:14,768::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:14,769::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:14,769::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:14,770::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:14,770::task::1185::TaskManager.Task::(prepare) Task=`14e1e225-744d-41b8-a1d7-d6a25aa4a4d4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:14,770::task::595::TaskManager.Task::(_updateState) Task=`14e1e225-744d-41b8-a1d7-d6a25aa4a4d4`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:14,771::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:14,771::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:14,771::task::990::TaskManager.Task::(_decref) Task=`14e1e225-744d-41b8-a1d7-d6a25aa4a4d4`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:15,776::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:15,777::task::595::TaskManager.Task::(_updateState) Task=`279cbc80-40bc-44e3-b4af-ff4b39e48aef`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:15,777::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:15,777::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:15,777::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:15,778::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:15,778::task::1185::TaskManager.Task::(prepare) Task=`279cbc80-40bc-44e3-b4af-ff4b39e48aef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:15,778::task::595::TaskManager.Task::(_updateState) Task=`279cbc80-40bc-44e3-b4af-ff4b39e48aef`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:15,779::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:15,779::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:15,779::task::990::TaskManager.Task::(_decref) Task=`279cbc80-40bc-44e3-b4af-ff4b39e48aef`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:59:16,691::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:59:16,692::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:59:16,710::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:59:16,711::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:59:16,724::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:59:16,725::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:59:16,725::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:59:16,726::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:59:16,786::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:16,788::task::595::TaskManager.Task::(_updateState) Task=`c51d021a-d6c1-452a-a715-1e5980a5854b`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:16,789::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:16,789::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:16,790::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:16,790::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:16,791::task::1185::TaskManager.Task::(prepare) Task=`c51d021a-d6c1-452a-a715-1e5980a5854b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:16,791::task::595::TaskManager.Task::(_updateState) Task=`c51d021a-d6c1-452a-a715-1e5980a5854b`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:16,792::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:16,792::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:16,792::task::990::TaskManager.Task::(_decref) Task=`c51d021a-d6c1-452a-a715-1e5980a5854b`::ref 0 aborting False
>Thread-39::DEBUG::2014-04-11 22:59:16,983::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:59:16,984::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-33::DEBUG::2014-04-11 22:59:17,025::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-39::ERROR::2014-04-11 22:59:17,029::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:59:17,033::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::WARNING::2014-04-11 22:59:17,037::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:59:17,038::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:59:17,069::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:59:17,069::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:59:17,798::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:17,799::task::595::TaskManager.Task::(_updateState) Task=`c437fb59-7997-4c31-8669-72eee0cb4d59`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:17,799::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:17,800::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:17,800::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:17,800::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:17,801::task::1185::TaskManager.Task::(prepare) Task=`c437fb59-7997-4c31-8669-72eee0cb4d59`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:17,801::task::595::TaskManager.Task::(_updateState) Task=`c437fb59-7997-4c31-8669-72eee0cb4d59`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:17,801::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:17,802::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:17,802::task::990::TaskManager.Task::(_decref) Task=`c437fb59-7997-4c31-8669-72eee0cb4d59`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:18,808::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:18,809::task::595::TaskManager.Task::(_updateState) Task=`f57d51d3-47f7-4771-b6b9-03deacb0a4d0`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:18,810::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:18,810::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:18,810::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:18,811::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:18,811::task::1185::TaskManager.Task::(prepare) Task=`f57d51d3-47f7-4771-b6b9-03deacb0a4d0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:18,811::task::595::TaskManager.Task::(_updateState) Task=`f57d51d3-47f7-4771-b6b9-03deacb0a4d0`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:18,811::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:18,812::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:18,812::task::990::TaskManager.Task::(_decref) Task=`f57d51d3-47f7-4771-b6b9-03deacb0a4d0`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:19,817::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:19,817::task::595::TaskManager.Task::(_updateState) Task=`8a3f0133-36a9-41b4-8b5e-d4a3d2de03e7`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:19,818::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:19,818::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:19,818::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:19,819::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:19,819::task::1185::TaskManager.Task::(prepare) Task=`8a3f0133-36a9-41b4-8b5e-d4a3d2de03e7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:19,819::task::595::TaskManager.Task::(_updateState) Task=`8a3f0133-36a9-41b4-8b5e-d4a3d2de03e7`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:19,820::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:19,820::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:19,820::task::990::TaskManager.Task::(_decref) Task=`8a3f0133-36a9-41b4-8b5e-d4a3d2de03e7`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:20,825::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:20,825::task::595::TaskManager.Task::(_updateState) Task=`67f44e0d-6f62-4b4a-a1a6-403e89c5c489`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:20,826::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:20,826::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:20,826::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:20,827::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:20,827::task::1185::TaskManager.Task::(prepare) Task=`67f44e0d-6f62-4b4a-a1a6-403e89c5c489`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:20,827::task::595::TaskManager.Task::(_updateState) Task=`67f44e0d-6f62-4b4a-a1a6-403e89c5c489`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:20,827::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:20,828::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:20,828::task::990::TaskManager.Task::(_decref) Task=`67f44e0d-6f62-4b4a-a1a6-403e89c5c489`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:21,833::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:21,846::task::595::TaskManager.Task::(_updateState) Task=`d720b8b8-4ba5-4f13-a0de-2e7768c3909b`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:21,846::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:21,847::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:21,847::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:21,847::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:21,847::task::1185::TaskManager.Task::(prepare) Task=`d720b8b8-4ba5-4f13-a0de-2e7768c3909b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:21,848::task::595::TaskManager.Task::(_updateState) Task=`d720b8b8-4ba5-4f13-a0de-2e7768c3909b`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:21,848::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:21,848::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:21,849::task::990::TaskManager.Task::(_decref) Task=`d720b8b8-4ba5-4f13-a0de-2e7768c3909b`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:21,928::task::595::TaskManager.Task::(_updateState) Task=`4d955a1b-6071-4000-aff4-486c9d97de17`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:21,929::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:59:21,929::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00048539', 'lastCheck': '8.6', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413842', 'lastCheck': '8.7', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.9', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000552306', 'lastCheck': '8.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '110.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000348219', 'lastCheck': '8.1', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:59:21,930::task::1185::TaskManager.Task::(prepare) Task=`4d955a1b-6071-4000-aff4-486c9d97de17`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00048539', 'lastCheck': '8.6', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413842', 'lastCheck': '8.7', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.9', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000552306', 'lastCheck': '8.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '110.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000348219', 'lastCheck': '8.1', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:59:21,930::task::595::TaskManager.Task::(_updateState) Task=`4d955a1b-6071-4000-aff4-486c9d97de17`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:21,930::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:21,931::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:21,931::task::990::TaskManager.Task::(_decref) Task=`4d955a1b-6071-4000-aff4-486c9d97de17`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:22,854::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:22,854::task::595::TaskManager.Task::(_updateState) Task=`12a63371-6eae-4512-881b-368421051cc6`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:22,855::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:22,855::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:22,855::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:22,856::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:22,856::task::1185::TaskManager.Task::(prepare) Task=`12a63371-6eae-4512-881b-368421051cc6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:22,856::task::595::TaskManager.Task::(_updateState) Task=`12a63371-6eae-4512-881b-368421051cc6`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:22,856::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:22,857::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:22,857::task::990::TaskManager.Task::(_decref) Task=`12a63371-6eae-4512-881b-368421051cc6`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:59:23,226::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:59:23,239::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000335394 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:59:23,372::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:59:23,384::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000413523 s, 1.7 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:59:23,769::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:59:23,783::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000492008 s, 1.5 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:59:23,824::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:59:23,836::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000364182 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:59:23,863::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:23,864::task::595::TaskManager.Task::(_updateState) Task=`64412dc9-88a6-439d-82c2-70c07e0bd7eb`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:23,864::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:23,864::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:23,865::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:23,865::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:23,865::task::1185::TaskManager.Task::(prepare) Task=`64412dc9-88a6-439d-82c2-70c07e0bd7eb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:23,866::task::595::TaskManager.Task::(_updateState) Task=`64412dc9-88a6-439d-82c2-70c07e0bd7eb`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:23,866::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:23,867::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:23,867::task::990::TaskManager.Task::(_decref) Task=`64412dc9-88a6-439d-82c2-70c07e0bd7eb`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:24,872::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:24,873::task::595::TaskManager.Task::(_updateState) Task=`56697163-1bab-41e3-b292-a435847c8317`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:24,873::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:24,873::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:24,874::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:24,874::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:24,874::task::1185::TaskManager.Task::(prepare) Task=`56697163-1bab-41e3-b292-a435847c8317`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:24,875::task::595::TaskManager.Task::(_updateState) Task=`56697163-1bab-41e3-b292-a435847c8317`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:24,875::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:24,875::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:24,875::task::990::TaskManager.Task::(_decref) Task=`56697163-1bab-41e3-b292-a435847c8317`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:25,880::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:25,881::task::595::TaskManager.Task::(_updateState) Task=`c0b33738-54be-4791-af29-5b952bc88171`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:25,881::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:25,881::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:25,882::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:25,882::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:25,882::task::1185::TaskManager.Task::(prepare) Task=`c0b33738-54be-4791-af29-5b952bc88171`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:25,883::task::595::TaskManager.Task::(_updateState) Task=`c0b33738-54be-4791-af29-5b952bc88171`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:25,883::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:25,883::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:25,884::task::990::TaskManager.Task::(_decref) Task=`c0b33738-54be-4791-af29-5b952bc88171`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:26,888::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:26,891::task::595::TaskManager.Task::(_updateState) Task=`291748f2-e871-484c-a1d2-e2bb7d8faf7c`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:26,892::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:26,892::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:26,892::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:26,892::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:26,893::task::1185::TaskManager.Task::(prepare) Task=`291748f2-e871-484c-a1d2-e2bb7d8faf7c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:26,893::task::595::TaskManager.Task::(_updateState) Task=`291748f2-e871-484c-a1d2-e2bb7d8faf7c`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:26,893::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:26,893::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:26,894::task::990::TaskManager.Task::(_decref) Task=`291748f2-e871-484c-a1d2-e2bb7d8faf7c`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:59:27,038::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:59:27,038::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:59:27,039::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:59:27,039::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:59:27,071::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:59:27,071::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:59:27,072::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:59:27,073::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:59:27,308::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:59:27,311::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:59:27,344::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:59:27,347::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:59:27,353::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:59:27,355::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:59:27,355::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:59:27,384::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:59:27,385::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:59:27,899::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:27,900::task::595::TaskManager.Task::(_updateState) Task=`04ce6a26-8f41-4003-96dc-d4a7528ef197`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:27,901::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:27,901::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:27,901::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:27,901::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:27,902::task::1185::TaskManager.Task::(prepare) Task=`04ce6a26-8f41-4003-96dc-d4a7528ef197`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:27,902::task::595::TaskManager.Task::(_updateState) Task=`04ce6a26-8f41-4003-96dc-d4a7528ef197`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:27,902::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:27,903::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:27,903::task::990::TaskManager.Task::(_decref) Task=`04ce6a26-8f41-4003-96dc-d4a7528ef197`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:28,909::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:28,910::task::595::TaskManager.Task::(_updateState) Task=`995728e3-469d-474e-910c-a1bd25851cdc`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:28,911::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:28,911::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:28,911::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:28,912::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:28,912::task::1185::TaskManager.Task::(prepare) Task=`995728e3-469d-474e-910c-a1bd25851cdc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:28,912::task::595::TaskManager.Task::(_updateState) Task=`995728e3-469d-474e-910c-a1bd25851cdc`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:28,913::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:28,913::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:28,913::task::990::TaskManager.Task::(_decref) Task=`995728e3-469d-474e-910c-a1bd25851cdc`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:29,918::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:29,919::task::595::TaskManager.Task::(_updateState) Task=`62e523e6-f376-44de-91ef-46cabfa2baeb`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:29,920::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:29,920::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:29,920::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:29,920::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:29,921::task::1185::TaskManager.Task::(prepare) Task=`62e523e6-f376-44de-91ef-46cabfa2baeb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:29,921::task::595::TaskManager.Task::(_updateState) Task=`62e523e6-f376-44de-91ef-46cabfa2baeb`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:29,921::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:29,922::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:29,922::task::990::TaskManager.Task::(_decref) Task=`62e523e6-f376-44de-91ef-46cabfa2baeb`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:30,927::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:30,927::task::595::TaskManager.Task::(_updateState) Task=`8687d34c-140c-441a-9b55-2ac74176d704`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:30,928::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:30,928::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:30,928::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:30,929::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:30,929::task::1185::TaskManager.Task::(prepare) Task=`8687d34c-140c-441a-9b55-2ac74176d704`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:30,929::task::595::TaskManager.Task::(_updateState) Task=`8687d34c-140c-441a-9b55-2ac74176d704`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:30,929::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:30,930::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:30,930::task::990::TaskManager.Task::(_decref) Task=`8687d34c-140c-441a-9b55-2ac74176d704`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:31,935::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:31,935::task::595::TaskManager.Task::(_updateState) Task=`b2ea4ad3-ff81-4917-9a6c-6b6b67cc0930`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:31,935::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:31,936::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:31,936::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:31,936::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:31,937::task::1185::TaskManager.Task::(prepare) Task=`b2ea4ad3-ff81-4917-9a6c-6b6b67cc0930`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:31,937::task::595::TaskManager.Task::(_updateState) Task=`b2ea4ad3-ff81-4917-9a6c-6b6b67cc0930`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:31,937::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:31,937::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:31,938::task::990::TaskManager.Task::(_decref) Task=`b2ea4ad3-ff81-4917-9a6c-6b6b67cc0930`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:32,942::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:32,943::task::595::TaskManager.Task::(_updateState) Task=`b2f8b1aa-a2ce-42ed-a622-1222b94d82ca`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:32,943::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:32,944::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:32,944::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:32,944::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:32,944::task::1185::TaskManager.Task::(prepare) Task=`b2f8b1aa-a2ce-42ed-a622-1222b94d82ca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:32,945::task::595::TaskManager.Task::(_updateState) Task=`b2f8b1aa-a2ce-42ed-a622-1222b94d82ca`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:32,945::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:32,945::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:32,946::task::990::TaskManager.Task::(_decref) Task=`b2f8b1aa-a2ce-42ed-a622-1222b94d82ca`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:59:33,264::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:59:33,277::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000405978 s, 909 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:59:33,414::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:59:33,427::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000499164 s, 1.4 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:59:33,809::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:59:33,822::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000487717 s, 1.5 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:59:33,866::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:59:33,878::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000450471 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:59:33,951::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:33,952::task::595::TaskManager.Task::(_updateState) Task=`748051c5-d97d-442b-810c-d93447a592b7`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:33,953::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:33,953::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:33,953::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:33,954::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:33,954::task::1185::TaskManager.Task::(prepare) Task=`748051c5-d97d-442b-810c-d93447a592b7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:33,954::task::595::TaskManager.Task::(_updateState) Task=`748051c5-d97d-442b-810c-d93447a592b7`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:33,955::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:33,955::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:33,955::task::990::TaskManager.Task::(_decref) Task=`748051c5-d97d-442b-810c-d93447a592b7`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:34,961::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:34,962::task::595::TaskManager.Task::(_updateState) Task=`0331cdf6-edf9-4a0a-a4b1-679b0b7483fc`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:34,963::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:34,963::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:34,963::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:34,964::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:34,964::task::1185::TaskManager.Task::(prepare) Task=`0331cdf6-edf9-4a0a-a4b1-679b0b7483fc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:34,964::task::595::TaskManager.Task::(_updateState) Task=`0331cdf6-edf9-4a0a-a4b1-679b0b7483fc`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:34,965::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:34,965::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:34,965::task::990::TaskManager.Task::(_decref) Task=`0331cdf6-edf9-4a0a-a4b1-679b0b7483fc`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:35,971::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:35,971::task::595::TaskManager.Task::(_updateState) Task=`cd76867b-af37-4672-bcb9-9e9788d3a955`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:35,972::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:35,972::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:35,972::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:35,973::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:35,973::task::1185::TaskManager.Task::(prepare) Task=`cd76867b-af37-4672-bcb9-9e9788d3a955`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:35,974::task::595::TaskManager.Task::(_updateState) Task=`cd76867b-af37-4672-bcb9-9e9788d3a955`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:35,974::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:35,974::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:35,975::task::990::TaskManager.Task::(_decref) Task=`cd76867b-af37-4672-bcb9-9e9788d3a955`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:36,980::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:36,980::task::595::TaskManager.Task::(_updateState) Task=`73018eec-c27b-4d31-b964-a1bad22dae80`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:36,981::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:36,981::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:36,981::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:36,981::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:36,982::task::1185::TaskManager.Task::(prepare) Task=`73018eec-c27b-4d31-b964-a1bad22dae80`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:36,982::task::595::TaskManager.Task::(_updateState) Task=`73018eec-c27b-4d31-b964-a1bad22dae80`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:36,982::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:36,983::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:36,983::task::990::TaskManager.Task::(_decref) Task=`73018eec-c27b-4d31-b964-a1bad22dae80`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:37,308::task::595::TaskManager.Task::(_updateState) Task=`1952d4a0-f9ac-4aa7-9239-400326d6856d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:37,308::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:59:37,309::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000499164', 'lastCheck': '3.9', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000405978', 'lastCheck': '4.0', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.9', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000487717', 'lastCheck': '3.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.0', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '126.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000450471', 'lastCheck': '3.4', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:59:37,310::task::1185::TaskManager.Task::(prepare) Task=`1952d4a0-f9ac-4aa7-9239-400326d6856d`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000499164', 'lastCheck': '3.9', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000405978', 'lastCheck': '4.0', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.9', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000487717', 'lastCheck': '3.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.0', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '126.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000450471', 'lastCheck': '3.4', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:59:37,310::task::595::TaskManager.Task::(_updateState) Task=`1952d4a0-f9ac-4aa7-9239-400326d6856d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:37,310::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:37,311::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:37,311::task::990::TaskManager.Task::(_decref) Task=`1952d4a0-f9ac-4aa7-9239-400326d6856d`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:59:37,353::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:59:37,353::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:59:37,354::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:59:37,354::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:59:37,386::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:59:37,387::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:59:37,387::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:59:37,388::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:59:37,623::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:59:37,626::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::ERROR::2014-04-11 22:59:37,661::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:59:37,661::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-33::DEBUG::2014-04-11 22:59:37,667::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:59:37,669::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::DEBUG::2014-04-11 22:59:37,670::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:59:37,701::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:59:37,701::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 22:59:37,989::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:38,011::task::595::TaskManager.Task::(_updateState) Task=`4391fc55-397d-4251-9544-df4b0502acf1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:38,012::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:38,012::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:38,012::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:38,013::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:38,013::task::1185::TaskManager.Task::(prepare) Task=`4391fc55-397d-4251-9544-df4b0502acf1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:38,013::task::595::TaskManager.Task::(_updateState) Task=`4391fc55-397d-4251-9544-df4b0502acf1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:38,014::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:38,014::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:38,014::task::990::TaskManager.Task::(_decref) Task=`4391fc55-397d-4251-9544-df4b0502acf1`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:39,031::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:39,032::task::595::TaskManager.Task::(_updateState) Task=`1142807a-1b4b-4159-9a0a-87a0b29a5b95`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:39,032::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:39,033::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:39,033::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:39,033::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:39,034::task::1185::TaskManager.Task::(prepare) Task=`1142807a-1b4b-4159-9a0a-87a0b29a5b95`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:39,034::task::595::TaskManager.Task::(_updateState) Task=`1142807a-1b4b-4159-9a0a-87a0b29a5b95`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:39,034::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:39,035::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:39,035::task::990::TaskManager.Task::(_decref) Task=`1142807a-1b4b-4159-9a0a-87a0b29a5b95`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:40,041::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:40,041::task::595::TaskManager.Task::(_updateState) Task=`b9acd7d8-aef1-4296-8952-a75fefe8ab70`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:40,042::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:40,042::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:40,042::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:40,042::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:40,043::task::1185::TaskManager.Task::(prepare) Task=`b9acd7d8-aef1-4296-8952-a75fefe8ab70`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:40,043::task::595::TaskManager.Task::(_updateState) Task=`b9acd7d8-aef1-4296-8952-a75fefe8ab70`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:40,043::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:40,044::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:40,044::task::990::TaskManager.Task::(_decref) Task=`b9acd7d8-aef1-4296-8952-a75fefe8ab70`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:41,048::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:41,049::task::595::TaskManager.Task::(_updateState) Task=`16f2f6ed-6f15-4069-96f4-b6c480941d10`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:41,049::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:41,050::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:41,050::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:41,050::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:41,051::task::1185::TaskManager.Task::(prepare) Task=`16f2f6ed-6f15-4069-96f4-b6c480941d10`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:41,051::task::595::TaskManager.Task::(_updateState) Task=`16f2f6ed-6f15-4069-96f4-b6c480941d10`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:41,051::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:41,051::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:41,052::task::990::TaskManager.Task::(_decref) Task=`16f2f6ed-6f15-4069-96f4-b6c480941d10`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:42,056::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:42,057::task::595::TaskManager.Task::(_updateState) Task=`61c14be9-3859-411a-9ce0-53f84cc9c669`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:42,057::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:42,058::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:42,058::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:42,058::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:42,059::task::1185::TaskManager.Task::(prepare) Task=`61c14be9-3859-411a-9ce0-53f84cc9c669`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:42,059::task::595::TaskManager.Task::(_updateState) Task=`61c14be9-3859-411a-9ce0-53f84cc9c669`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:42,059::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:42,059::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:42,060::task::990::TaskManager.Task::(_decref) Task=`61c14be9-3859-411a-9ce0-53f84cc9c669`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:43,064::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:43,081::task::595::TaskManager.Task::(_updateState) Task=`6253d6b8-7335-46a6-8acb-cb3bdec81ee6`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:43,082::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:43,082::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:43,082::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:43,082::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:43,083::task::1185::TaskManager.Task::(prepare) Task=`6253d6b8-7335-46a6-8acb-cb3bdec81ee6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:43,083::task::595::TaskManager.Task::(_updateState) Task=`6253d6b8-7335-46a6-8acb-cb3bdec81ee6`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:43,083::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:43,084::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:43,084::task::990::TaskManager.Task::(_decref) Task=`6253d6b8-7335-46a6-8acb-cb3bdec81ee6`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:59:43,302::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:59:43,315::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000346167 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:59:43,456::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:59:43,469::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000485038 s, 1.5 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:59:43,851::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:59:43,864::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00051056 s, 1.5 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:59:43,908::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:59:43,921::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000379966 s, 2.0 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:59:44,089::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:44,090::task::595::TaskManager.Task::(_updateState) Task=`6dc1a420-0dbf-4fe4-a1d3-bc68e2c4ee3a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:44,091::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:44,091::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:44,091::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:44,092::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:44,092::task::1185::TaskManager.Task::(prepare) Task=`6dc1a420-0dbf-4fe4-a1d3-bc68e2c4ee3a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:44,092::task::595::TaskManager.Task::(_updateState) Task=`6dc1a420-0dbf-4fe4-a1d3-bc68e2c4ee3a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:44,093::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:44,093::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:44,093::task::990::TaskManager.Task::(_decref) Task=`6dc1a420-0dbf-4fe4-a1d3-bc68e2c4ee3a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:45,099::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:45,100::task::595::TaskManager.Task::(_updateState) Task=`00a6dabd-201b-4c1a-83b1-37e248cbec82`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:45,100::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:45,100::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:45,101::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:45,101::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:45,101::task::1185::TaskManager.Task::(prepare) Task=`00a6dabd-201b-4c1a-83b1-37e248cbec82`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:45,102::task::595::TaskManager.Task::(_updateState) Task=`00a6dabd-201b-4c1a-83b1-37e248cbec82`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:45,102::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:45,102::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:45,103::task::990::TaskManager.Task::(_decref) Task=`00a6dabd-201b-4c1a-83b1-37e248cbec82`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:46,108::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:46,108::task::595::TaskManager.Task::(_updateState) Task=`d2fd0140-4ac3-419f-9d33-8b03e0bee4ca`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:46,109::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:46,109::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:46,109::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:46,109::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:46,110::task::1185::TaskManager.Task::(prepare) Task=`d2fd0140-4ac3-419f-9d33-8b03e0bee4ca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:46,110::task::595::TaskManager.Task::(_updateState) Task=`d2fd0140-4ac3-419f-9d33-8b03e0bee4ca`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:46,110::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:46,110::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:46,111::task::990::TaskManager.Task::(_decref) Task=`d2fd0140-4ac3-419f-9d33-8b03e0bee4ca`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:47,115::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:47,116::task::595::TaskManager.Task::(_updateState) Task=`3d0ccc1a-5ead-45a1-b0cb-0790c3a02d93`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:47,116::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:47,117::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:47,117::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:47,117::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:47,118::task::1185::TaskManager.Task::(prepare) Task=`3d0ccc1a-5ead-45a1-b0cb-0790c3a02d93`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:47,118::task::595::TaskManager.Task::(_updateState) Task=`3d0ccc1a-5ead-45a1-b0cb-0790c3a02d93`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:47,118::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:47,118::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:47,119::task::990::TaskManager.Task::(_decref) Task=`3d0ccc1a-5ead-45a1-b0cb-0790c3a02d93`::ref 0 aborting False
>Thread-39::ERROR::2014-04-11 22:59:47,662::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 22:59:47,663::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 22:59:47,663::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 22:59:47,664::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::ERROR::2014-04-11 22:59:47,703::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:59:47,703::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 22:59:47,704::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-33::DEBUG::2014-04-11 22:59:47,705::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:59:47,986::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-33::DEBUG::2014-04-11 22:59:47,994::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:59:47,997::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-39::WARNING::2014-04-11 22:59:47,996::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::DEBUG::2014-04-11 22:59:48,002::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:59:48,070::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:59:48,079::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-39::ERROR::2014-04-11 22:59:48,082::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:59:48,085::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-14::DEBUG::2014-04-11 22:59:48,124::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:48,125::task::595::TaskManager.Task::(_updateState) Task=`4dda0834-cc2f-447d-b93d-09ce58261a44`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:48,125::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:48,126::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:48,126::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:48,127::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:48,127::task::1185::TaskManager.Task::(prepare) Task=`4dda0834-cc2f-447d-b93d-09ce58261a44`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:48,127::task::595::TaskManager.Task::(_updateState) Task=`4dda0834-cc2f-447d-b93d-09ce58261a44`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:48,128::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:48,128::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:48,128::task::990::TaskManager.Task::(_decref) Task=`4dda0834-cc2f-447d-b93d-09ce58261a44`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:49,134::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:49,136::task::595::TaskManager.Task::(_updateState) Task=`9d9d6015-8d60-42f7-b801-7c17f8471407`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:49,137::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:49,137::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:49,137::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:49,138::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:49,138::task::1185::TaskManager.Task::(prepare) Task=`9d9d6015-8d60-42f7-b801-7c17f8471407`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:49,139::task::595::TaskManager.Task::(_updateState) Task=`9d9d6015-8d60-42f7-b801-7c17f8471407`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:49,139::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:49,139::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:49,140::task::990::TaskManager.Task::(_decref) Task=`9d9d6015-8d60-42f7-b801-7c17f8471407`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:50,145::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:50,145::task::595::TaskManager.Task::(_updateState) Task=`8545d396-ac07-4ab2-b049-062fda093a90`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:50,146::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:50,146::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:50,147::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:50,147::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:50,147::task::1185::TaskManager.Task::(prepare) Task=`8545d396-ac07-4ab2-b049-062fda093a90`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:50,148::task::595::TaskManager.Task::(_updateState) Task=`8545d396-ac07-4ab2-b049-062fda093a90`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:50,148::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:50,148::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:50,149::task::990::TaskManager.Task::(_decref) Task=`8545d396-ac07-4ab2-b049-062fda093a90`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:51,153::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:51,154::task::595::TaskManager.Task::(_updateState) Task=`59c8f34c-12f9-461c-97be-90e9f384c77b`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:51,154::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:51,155::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:51,155::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:51,155::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:51,156::task::1185::TaskManager.Task::(prepare) Task=`59c8f34c-12f9-461c-97be-90e9f384c77b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:51,156::task::595::TaskManager.Task::(_updateState) Task=`59c8f34c-12f9-461c-97be-90e9f384c77b`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:51,156::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:51,157::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:51,157::task::990::TaskManager.Task::(_decref) Task=`59c8f34c-12f9-461c-97be-90e9f384c77b`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:52,162::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:52,162::task::595::TaskManager.Task::(_updateState) Task=`6c5ec55e-4048-4016-a800-235ac621b30f`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:52,163::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:52,163::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:52,163::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:52,164::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:52,164::task::1185::TaskManager.Task::(prepare) Task=`6c5ec55e-4048-4016-a800-235ac621b30f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:52,164::task::595::TaskManager.Task::(_updateState) Task=`6c5ec55e-4048-4016-a800-235ac621b30f`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:52,164::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:52,165::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:52,165::task::990::TaskManager.Task::(_decref) Task=`6c5ec55e-4048-4016-a800-235ac621b30f`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:52,666::task::595::TaskManager.Task::(_updateState) Task=`9e19003d-90b9-4a2a-9ed9-da678d55463d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:52,667::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 22:59:52,667::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000485038', 'lastCheck': '9.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346167', 'lastCheck': '9.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00051056', 'lastCheck': '8.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.6', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '141.6', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000379966', 'lastCheck': '8.7', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:59:52,668::task::1185::TaskManager.Task::(prepare) Task=`9e19003d-90b9-4a2a-9ed9-da678d55463d`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000485038', 'lastCheck': '9.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346167', 'lastCheck': '9.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00051056', 'lastCheck': '8.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.6', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '141.6', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000379966', 'lastCheck': '8.7', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 22:59:52,668::task::595::TaskManager.Task::(_updateState) Task=`9e19003d-90b9-4a2a-9ed9-da678d55463d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:52,668::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:52,669::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:52,669::task::990::TaskManager.Task::(_decref) Task=`9e19003d-90b9-4a2a-9ed9-da678d55463d`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:53,170::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:53,171::task::595::TaskManager.Task::(_updateState) Task=`f09ca30a-efff-4a7e-8573-75520490eae2`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:53,171::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:53,172::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:53,172::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:53,172::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:53,173::task::1185::TaskManager.Task::(prepare) Task=`f09ca30a-efff-4a7e-8573-75520490eae2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:53,173::task::595::TaskManager.Task::(_updateState) Task=`f09ca30a-efff-4a7e-8573-75520490eae2`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:53,173::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:53,173::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:53,174::task::990::TaskManager.Task::(_decref) Task=`f09ca30a-efff-4a7e-8573-75520490eae2`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 22:59:53,340::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 22:59:53,354::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000327478 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 22:59:53,508::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 22:59:53,521::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.00051355 s, 1.4 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 22:59:53,893::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 22:59:53,905::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000341264 s, 2.2 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 22:59:53,952::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 22:59:53,964::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000320057 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 22:59:54,179::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:54,180::task::595::TaskManager.Task::(_updateState) Task=`6853ab5b-19bd-4cda-8a10-e812a4dc000a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:54,181::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:54,181::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:54,181::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:54,182::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:54,182::task::1185::TaskManager.Task::(prepare) Task=`6853ab5b-19bd-4cda-8a10-e812a4dc000a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:54,183::task::595::TaskManager.Task::(_updateState) Task=`6853ab5b-19bd-4cda-8a10-e812a4dc000a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:54,183::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:54,183::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:54,184::task::990::TaskManager.Task::(_decref) Task=`6853ab5b-19bd-4cda-8a10-e812a4dc000a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:55,189::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:55,189::task::595::TaskManager.Task::(_updateState) Task=`756ccd38-d47b-4ab7-bcfd-43db5b3a56f8`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:55,190::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:55,190::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:55,190::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:55,191::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:55,191::task::1185::TaskManager.Task::(prepare) Task=`756ccd38-d47b-4ab7-bcfd-43db5b3a56f8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:55,191::task::595::TaskManager.Task::(_updateState) Task=`756ccd38-d47b-4ab7-bcfd-43db5b3a56f8`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:55,191::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:55,192::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:55,192::task::990::TaskManager.Task::(_decref) Task=`756ccd38-d47b-4ab7-bcfd-43db5b3a56f8`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:56,196::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:56,197::task::595::TaskManager.Task::(_updateState) Task=`56c75602-64e2-445b-88aa-239c73ec458d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:56,197::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:56,197::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:56,198::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:56,198::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:56,198::task::1185::TaskManager.Task::(prepare) Task=`56c75602-64e2-445b-88aa-239c73ec458d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:56,199::task::595::TaskManager.Task::(_updateState) Task=`56c75602-64e2-445b-88aa-239c73ec458d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:56,199::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:56,199::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:56,199::task::990::TaskManager.Task::(_decref) Task=`56c75602-64e2-445b-88aa-239c73ec458d`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 22:59:57,204::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:57,205::task::595::TaskManager.Task::(_updateState) Task=`d3d31c35-155b-45e8-b88a-bafe62cda340`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:57,205::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:57,205::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:57,205::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:57,206::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:57,206::task::1185::TaskManager.Task::(prepare) Task=`d3d31c35-155b-45e8-b88a-bafe62cda340`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:57,206::task::595::TaskManager.Task::(_updateState) Task=`d3d31c35-155b-45e8-b88a-bafe62cda340`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:57,207::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:57,207::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:57,207::task::990::TaskManager.Task::(_decref) Task=`d3d31c35-155b-45e8-b88a-bafe62cda340`::ref 0 aborting False
>Thread-33::ERROR::2014-04-11 22:59:58,085::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 22:59:58,086::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-39::ERROR::2014-04-11 22:59:58,086::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-33::DEBUG::2014-04-11 22:59:58,095::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::ERROR::2014-04-11 22:59:58,096::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-33::DEBUG::2014-04-11 22:59:58,097::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::DEBUG::2014-04-11 22:59:58,098::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-39::DEBUG::2014-04-11 22:59:58,109::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-14::DEBUG::2014-04-11 22:59:58,213::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:58,214::task::595::TaskManager.Task::(_updateState) Task=`c29661c7-89a4-4bcf-bdf2-eb84c9f28e2f`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:58,214::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:58,215::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:58,215::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:58,215::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:58,216::task::1185::TaskManager.Task::(prepare) Task=`c29661c7-89a4-4bcf-bdf2-eb84c9f28e2f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:58,216::task::595::TaskManager.Task::(_updateState) Task=`c29661c7-89a4-4bcf-bdf2-eb84c9f28e2f`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:58,216::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:58,217::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:58,217::task::990::TaskManager.Task::(_decref) Task=`c29661c7-89a4-4bcf-bdf2-eb84c9f28e2f`::ref 0 aborting False
>Thread-33::DEBUG::2014-04-11 22:59:58,379::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 22:59:58,385::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-39::DEBUG::2014-04-11 22:59:58,388::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 22:59:58,402::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::DEBUG::2014-04-11 22:59:58,416::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-33::ERROR::2014-04-11 22:59:58,435::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 22:59:58,440::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-39::ERROR::2014-04-11 22:59:58,472::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 22:59:58,472::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-14::DEBUG::2014-04-11 22:59:59,222::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 22:59:59,223::task::595::TaskManager.Task::(_updateState) Task=`0e7992ee-dde7-4f3e-a2cc-938a30e9db87`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 22:59:59,224::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 22:59:59,224::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 22:59:59,224::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 22:59:59,225::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:59,225::task::1185::TaskManager.Task::(prepare) Task=`0e7992ee-dde7-4f3e-a2cc-938a30e9db87`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 22:59:59,225::task::595::TaskManager.Task::(_updateState) Task=`0e7992ee-dde7-4f3e-a2cc-938a30e9db87`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 22:59:59,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 22:59:59,226::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 22:59:59,226::task::990::TaskManager.Task::(_decref) Task=`0e7992ee-dde7-4f3e-a2cc-938a30e9db87`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:00,231::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:00,232::task::595::TaskManager.Task::(_updateState) Task=`ee17572b-fe3b-44fc-8c26-127e11ad87b1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:00,232::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:00,233::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:00,233::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:00,233::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:00,234::task::1185::TaskManager.Task::(prepare) Task=`ee17572b-fe3b-44fc-8c26-127e11ad87b1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:00,234::task::595::TaskManager.Task::(_updateState) Task=`ee17572b-fe3b-44fc-8c26-127e11ad87b1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:00,234::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:00,235::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:00,235::task::990::TaskManager.Task::(_decref) Task=`ee17572b-fe3b-44fc-8c26-127e11ad87b1`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:01,240::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:01,240::task::595::TaskManager.Task::(_updateState) Task=`49f2f467-8685-4f6f-abbb-a675acad16e8`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:01,241::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:01,241::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:01,241::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:01,241::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:01,242::task::1185::TaskManager.Task::(prepare) Task=`49f2f467-8685-4f6f-abbb-a675acad16e8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:01,242::task::595::TaskManager.Task::(_updateState) Task=`49f2f467-8685-4f6f-abbb-a675acad16e8`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:01,242::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:01,243::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:01,243::task::990::TaskManager.Task::(_decref) Task=`49f2f467-8685-4f6f-abbb-a675acad16e8`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:02,248::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:02,248::task::595::TaskManager.Task::(_updateState) Task=`34a529a1-14d2-4ccc-828f-9c67d7be1b59`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:02,249::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:02,249::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:02,249::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:02,250::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:02,250::task::1185::TaskManager.Task::(prepare) Task=`34a529a1-14d2-4ccc-828f-9c67d7be1b59`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:02,250::task::595::TaskManager.Task::(_updateState) Task=`34a529a1-14d2-4ccc-828f-9c67d7be1b59`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:02,251::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:02,251::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:02,251::task::990::TaskManager.Task::(_decref) Task=`34a529a1-14d2-4ccc-828f-9c67d7be1b59`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:03,256::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:03,257::task::595::TaskManager.Task::(_updateState) Task=`ddca807f-0b63-4845-be89-fed7eeb6e8fc`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:03,257::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:03,258::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:03,258::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:03,258::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:03,258::task::1185::TaskManager.Task::(prepare) Task=`ddca807f-0b63-4845-be89-fed7eeb6e8fc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:03,259::task::595::TaskManager.Task::(_updateState) Task=`ddca807f-0b63-4845-be89-fed7eeb6e8fc`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:03,259::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:03,259::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:03,260::task::990::TaskManager.Task::(_decref) Task=`ddca807f-0b63-4845-be89-fed7eeb6e8fc`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 23:00:03,381::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 23:00:03,398::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000343957 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 23:00:03,550::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 23:00:03,563::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000518859 s, 1.4 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 23:00:03,935::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 23:00:03,949::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000541906 s, 1.4 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 23:00:03,998::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 23:00:04,010::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00036034 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 23:00:04,265::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:04,266::task::595::TaskManager.Task::(_updateState) Task=`69a92bdc-708b-4e87-9f82-7dea14d3c9ba`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:04,267::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:04,267::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:04,267::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:04,268::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:04,268::task::1185::TaskManager.Task::(prepare) Task=`69a92bdc-708b-4e87-9f82-7dea14d3c9ba`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:04,268::task::595::TaskManager.Task::(_updateState) Task=`69a92bdc-708b-4e87-9f82-7dea14d3c9ba`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:04,269::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:04,269::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:04,269::task::990::TaskManager.Task::(_decref) Task=`69a92bdc-708b-4e87-9f82-7dea14d3c9ba`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:05,274::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:05,275::task::595::TaskManager.Task::(_updateState) Task=`36f5c837-a6fa-44fa-8645-c9351ea14568`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:05,275::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:05,276::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:05,276::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:05,276::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:05,277::task::1185::TaskManager.Task::(prepare) Task=`36f5c837-a6fa-44fa-8645-c9351ea14568`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:05,277::task::595::TaskManager.Task::(_updateState) Task=`36f5c837-a6fa-44fa-8645-c9351ea14568`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:05,277::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:05,278::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:05,278::task::990::TaskManager.Task::(_decref) Task=`36f5c837-a6fa-44fa-8645-c9351ea14568`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:06,283::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:06,283::task::595::TaskManager.Task::(_updateState) Task=`d16d726e-320b-44de-892c-ebba8afc5653`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:06,284::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:06,284::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:06,284::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:06,285::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:06,285::task::1185::TaskManager.Task::(prepare) Task=`d16d726e-320b-44de-892c-ebba8afc5653`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:06,285::task::595::TaskManager.Task::(_updateState) Task=`d16d726e-320b-44de-892c-ebba8afc5653`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:06,286::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:06,286::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:06,286::task::990::TaskManager.Task::(_decref) Task=`d16d726e-320b-44de-892c-ebba8afc5653`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:07,291::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:07,309::task::595::TaskManager.Task::(_updateState) Task=`a3cfe5ef-41bf-466f-8d68-ea0bf9b7d280`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:07,309::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:07,310::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:07,310::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:07,310::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:07,310::task::1185::TaskManager.Task::(prepare) Task=`a3cfe5ef-41bf-466f-8d68-ea0bf9b7d280`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:07,311::task::595::TaskManager.Task::(_updateState) Task=`a3cfe5ef-41bf-466f-8d68-ea0bf9b7d280`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:07,311::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:07,311::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:07,312::task::990::TaskManager.Task::(_decref) Task=`a3cfe5ef-41bf-466f-8d68-ea0bf9b7d280`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:07,983::task::595::TaskManager.Task::(_updateState) Task=`390e045a-9112-4ff3-bb06-6183abd87b5b`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:07,983::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 23:00:07,984::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000518859', 'lastCheck': '4.4', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000343957', 'lastCheck': '4.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.5', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000541906', 'lastCheck': '4.0', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '156.9', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00036034', 'lastCheck': '4.0', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:07,984::task::1185::TaskManager.Task::(prepare) Task=`390e045a-9112-4ff3-bb06-6183abd87b5b`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000518859', 'lastCheck': '4.4', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000343957', 'lastCheck': '4.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.5', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000541906', 'lastCheck': '4.0', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '156.9', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00036034', 'lastCheck': '4.0', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:07,985::task::595::TaskManager.Task::(_updateState) Task=`390e045a-9112-4ff3-bb06-6183abd87b5b`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:07,985::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:07,985::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:07,986::task::990::TaskManager.Task::(_decref) Task=`390e045a-9112-4ff3-bb06-6183abd87b5b`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:08,316::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:08,317::task::595::TaskManager.Task::(_updateState) Task=`1741d324-ae09-40d0-95ab-fdffd3e7f538`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:08,317::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:08,317::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:08,318::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:08,318::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:08,318::task::1185::TaskManager.Task::(prepare) Task=`1741d324-ae09-40d0-95ab-fdffd3e7f538`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:08,319::task::595::TaskManager.Task::(_updateState) Task=`1741d324-ae09-40d0-95ab-fdffd3e7f538`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:08,319::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:08,319::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:08,320::task::990::TaskManager.Task::(_decref) Task=`1741d324-ae09-40d0-95ab-fdffd3e7f538`::ref 0 aborting False
>Thread-33::ERROR::2014-04-11 23:00:08,454::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 23:00:08,455::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 23:00:08,455::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-33::DEBUG::2014-04-11 23:00:08,456::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::ERROR::2014-04-11 23:00:08,474::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 23:00:08,475::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 23:00:08,475::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-39::DEBUG::2014-04-11 23:00:08,476::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::DEBUG::2014-04-11 23:00:08,741::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 23:00:08,744::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::ERROR::2014-04-11 23:00:08,782::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-39::DEBUG::2014-04-11 23:00:08,783::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-33::ERROR::2014-04-11 23:00:08,786::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-39::WARNING::2014-04-11 23:00:08,792::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::DEBUG::2014-04-11 23:00:08,793::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-39::ERROR::2014-04-11 23:00:08,823::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 23:00:08,824::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-14::DEBUG::2014-04-11 23:00:09,325::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:09,326::task::595::TaskManager.Task::(_updateState) Task=`4adc83bc-3a72-4cd9-a843-4bf622cd7eaa`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:09,326::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:09,327::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:09,327::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:09,328::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:09,328::task::1185::TaskManager.Task::(prepare) Task=`4adc83bc-3a72-4cd9-a843-4bf622cd7eaa`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:09,328::task::595::TaskManager.Task::(_updateState) Task=`4adc83bc-3a72-4cd9-a843-4bf622cd7eaa`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:09,329::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:09,329::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:09,329::task::990::TaskManager.Task::(_decref) Task=`4adc83bc-3a72-4cd9-a843-4bf622cd7eaa`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:10,334::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:10,335::task::595::TaskManager.Task::(_updateState) Task=`c1382639-ecb5-4d5d-8318-64a9d652f897`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:10,335::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:10,336::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:10,336::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:10,336::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:10,336::task::1185::TaskManager.Task::(prepare) Task=`c1382639-ecb5-4d5d-8318-64a9d652f897`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:10,337::task::595::TaskManager.Task::(_updateState) Task=`c1382639-ecb5-4d5d-8318-64a9d652f897`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:10,337::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:10,337::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:10,338::task::990::TaskManager.Task::(_decref) Task=`c1382639-ecb5-4d5d-8318-64a9d652f897`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:11,342::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:11,342::task::595::TaskManager.Task::(_updateState) Task=`5ef7a2b0-85c0-4b1c-bfad-e0ebfefa0efc`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:11,343::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:11,343::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:11,343::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:11,344::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:11,344::task::1185::TaskManager.Task::(prepare) Task=`5ef7a2b0-85c0-4b1c-bfad-e0ebfefa0efc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:11,344::task::595::TaskManager.Task::(_updateState) Task=`5ef7a2b0-85c0-4b1c-bfad-e0ebfefa0efc`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:11,345::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:11,345::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:11,345::task::990::TaskManager.Task::(_decref) Task=`5ef7a2b0-85c0-4b1c-bfad-e0ebfefa0efc`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:12,349::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:12,350::task::595::TaskManager.Task::(_updateState) Task=`8c4048c4-8e95-41e7-9c0c-8893aaa1e9d3`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:12,350::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:12,351::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:12,351::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:12,351::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:12,352::task::1185::TaskManager.Task::(prepare) Task=`8c4048c4-8e95-41e7-9c0c-8893aaa1e9d3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:12,352::task::595::TaskManager.Task::(_updateState) Task=`8c4048c4-8e95-41e7-9c0c-8893aaa1e9d3`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:12,352::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:12,352::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:12,353::task::990::TaskManager.Task::(_decref) Task=`8c4048c4-8e95-41e7-9c0c-8893aaa1e9d3`::ref 0 aborting False
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,624::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:12,625::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,677::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,677::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,693::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,694::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,694::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,695::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,735::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,735::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`2ee18c70-c625-479f-81c3-1b070c2f8207`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,736::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,736::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,737::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`2ee18c70-c625-479f-81c3-1b070c2f8207`::Granted request
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:12,737::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a`
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,738::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`59a48651-840c-4a4f-951b-ba07b08418b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,738::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,739::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,739::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`59a48651-840c-4a4f-951b-ba07b08418b4`::Granted request
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,754::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,756::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,772::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,774::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:12,774::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:12,775::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,775::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,775::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,776::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,776::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,776::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,791::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,792::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,792::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
>Thread-342::DEBUG::2014-04-11 23:00:12,793::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-343::DEBUG::2014-04-11 23:00:12,794::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-342::DEBUG::2014-04-11 23:00:12,795::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`d1b465b3-aa93-4fbc-9e01-29b5137037c1`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-344::DEBUG::2014-04-11 23:00:12,795::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-343::DEBUG::2014-04-11 23:00:12,797::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`4a2d70c9-ad7f-4e76-8f8a-fcf08802bacc`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-345::ERROR::2014-04-11 23:00:12,797::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-342::DEBUG::2014-04-11 23:00:12,799::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive'
>Thread-346::ERROR::2014-04-11 23:00:12,799::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-344::DEBUG::2014-04-11 23:00:12,800::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`41ff77b2-310a-40a2-94bd-071757ba5a3b`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-343::DEBUG::2014-04-11 23:00:12,800::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive'
>Thread-345::ERROR::2014-04-11 23:00:12,801::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-347::DEBUG::2014-04-11 23:00:12,801::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 21619c8e-99ea-4813-be02-d708971e5393
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,802::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-342::DEBUG::2014-04-11 23:00:12,802::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user)
>Thread-346::ERROR::2014-04-11 23:00:12,802::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-344::DEBUG::2014-04-11 23:00:12,803::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive'
>Thread-345::DEBUG::2014-04-11 23:00:12,804::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-347::DEBUG::2014-04-11 23:00:12,805::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`8cd2444d-b94a-42a5-9d29-3e26eac098f0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-342::DEBUG::2014-04-11 23:00:12,806::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`d1b465b3-aa93-4fbc-9e01-29b5137037c1`::Granted request
>Thread-345::DEBUG::2014-04-11 23:00:12,809::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-346::DEBUG::2014-04-11 23:00:12,809::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-347::DEBUG::2014-04-11 23:00:12,810::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive'
>Thread-343::DEBUG::2014-04-11 23:00:12,810::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user)
>Thread-342::DEBUG::2014-04-11 23:00:12,823::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`b0f5bd84-921a-42a1-b858-7cd3acebd1fe`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-346::DEBUG::2014-04-11 23:00:12,825::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-343::DEBUG::2014-04-11 23:00:12,827::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`4a2d70c9-ad7f-4e76-8f8a-fcf08802bacc`::Granted request
>Thread-342::DEBUG::2014-04-11 23:00:12,827::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,839::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-343::DEBUG::2014-04-11 23:00:12,842::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`2f87ab4b-0ec3-4813-b7d3-3b2543939062`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,842::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-343::DEBUG::2014-04-11 23:00:12,843::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,843::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-344::DEBUG::2014-04-11 23:00:12,844::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user)
>Thread-344::DEBUG::2014-04-11 23:00:12,846::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`41ff77b2-310a-40a2-94bd-071757ba5a3b`::Granted request
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,847::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms
>Thread-347::DEBUG::2014-04-11 23:00:12,847::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user)
>Thread-344::DEBUG::2014-04-11 23:00:12,850::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`371c311f-b46b-4fde-a322-1ee4982d7eb4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-347::DEBUG::2014-04-11 23:00:12,850::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`8cd2444d-b94a-42a5-9d29-3e26eac098f0`::Granted request
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,851::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks
>Thread-344::DEBUG::2014-04-11 23:00:12,851::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive'
>Thread-342::DEBUG::2014-04-11 23:00:12,852::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user)
>Thread-347::DEBUG::2014-04-11 23:00:12,855::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`213cfa65-acb1-4cf4-9871-4e4fe811bb04`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-342::DEBUG::2014-04-11 23:00:12,855::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`b0f5bd84-921a-42a1-b858-7cd3acebd1fe`::Granted request
>Thread-347::DEBUG::2014-04-11 23:00:12,856::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive'
>Thread-343::DEBUG::2014-04-11 23:00:12,857::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user)
>Thread-343::DEBUG::2014-04-11 23:00:12,859::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`2f87ab4b-0ec3-4813-b7d3-3b2543939062`::Granted request
>Thread-344::DEBUG::2014-04-11 23:00:12,859::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user)
>Thread-343::DEBUG::2014-04-11 23:00:12,860::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` is not a data domain it is an 2 domain, not upgrading
>Thread-344::DEBUG::2014-04-11 23:00:12,860::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`371c311f-b46b-4fde-a322-1ee4982d7eb4`::Granted request
>Thread-343::DEBUG::2014-04-11 23:00:12,861::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87'
>Thread-347::DEBUG::2014-04-11 23:00:12,862::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user)
>Thread-347::DEBUG::2014-04-11 23:00:12,864::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`213cfa65-acb1-4cf4-9871-4e4fe811bb04`::Granted request
>Thread-343::DEBUG::2014-04-11 23:00:12,868::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users)
>Thread-343::DEBUG::2014-04-11 23:00:12,869::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it.
>Thread-343::DEBUG::2014-04-11 23:00:12,869::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records.
>Thread-343::DEBUG::2014-04-11 23:00:12,870::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87'
>Thread-343::DEBUG::2014-04-11 23:00:12,870::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users)
>Thread-343::DEBUG::2014-04-11 23:00:12,871::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it.
>Thread-343::DEBUG::2014-04-11 23:00:12,871::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,872::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>Thread-344::DEBUG::2014-04-11 23:00:12,881::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40']
>Thread-344::DEBUG::2014-04-11 23:00:12,881::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce'
>Thread-344::DEBUG::2014-04-11 23:00:12,882::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users)
>Thread-344::DEBUG::2014-04-11 23:00:12,882::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it.
>Thread-344::DEBUG::2014-04-11 23:00:12,882::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records.
>Thread-344::DEBUG::2014-04-11 23:00:12,883::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce'
>Thread-344::DEBUG::2014-04-11 23:00:12,883::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users)
>Thread-344::DEBUG::2014-04-11 23:00:12,883::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it.
>Thread-344::DEBUG::2014-04-11 23:00:12,884::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records.
>Thread-342::DEBUG::2014-04-11 23:00:12,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955']
>Thread-342::DEBUG::2014-04-11 23:00:12,889::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-342::DEBUG::2014-04-11 23:00:12,889::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-342::DEBUG::2014-04-11 23:00:12,890::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-342::DEBUG::2014-04-11 23:00:12,890::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-342::DEBUG::2014-04-11 23:00:12,890::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-342::DEBUG::2014-04-11 23:00:12,891::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-342::DEBUG::2014-04-11 23:00:12,891::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-342::DEBUG::2014-04-11 23:00:12,891::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-347::DEBUG::2014-04-11 23:00:12,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2']
>Thread-347::DEBUG::2014-04-11 23:00:12,895::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393'
>Thread-347::DEBUG::2014-04-11 23:00:12,896::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' (0 active users)
>Thread-347::DEBUG::2014-04-11 23:00:12,896::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it.
>Thread-347::DEBUG::2014-04-11 23:00:12,896::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393', Clearing records.
>Thread-347::DEBUG::2014-04-11 23:00:12,897::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393'
>Thread-347::DEBUG::2014-04-11 23:00:12,897::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' (0 active users)
>Thread-347::DEBUG::2014-04-11 23:00:12,897::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it.
>Thread-347::DEBUG::2014-04-11 23:00:12,897::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393', Clearing records.
>Thread-345::DEBUG::2014-04-11 23:00:13,100::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-345::WARNING::2014-04-11 23:00:13,104::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-346::DEBUG::2014-04-11 23:00:13,143::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-345::ERROR::2014-04-11 23:00:13,143::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,161::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,164::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-346::WARNING::2014-04-11 23:00:13,165::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,167::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,192::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-346::ERROR::2014-04-11 23:00:13,229::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-14::DEBUG::2014-04-11 23:00:13,358::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:13,359::task::595::TaskManager.Task::(_updateState) Task=`b25a3033-503b-4b1f-bb94-79cd05384430`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:13,360::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:13,360::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:13,360::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:13,361::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:13,361::task::1185::TaskManager.Task::(prepare) Task=`b25a3033-503b-4b1f-bb94-79cd05384430`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:13,361::task::595::TaskManager.Task::(_updateState) Task=`b25a3033-503b-4b1f-bb94-79cd05384430`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:13,362::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:13,362::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:13,362::task::990::TaskManager.Task::(_decref) Task=`b25a3033-503b-4b1f-bb94-79cd05384430`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 23:00:13,426::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 23:00:13,439::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000322894 s, 1.1 MB/s\n'; <rc> = 0
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,476::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::WARNING::2014-04-11 23:00:13,478::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,479::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,510::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,510::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
>    self.checkBackupDomain(__securityOverride=True)
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
>    dom = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,526::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,526::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,526::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,528::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-22::DEBUG::2014-04-11 23:00:13,596::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 23:00:13,610::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000512307 s, 1.4 MB/s\n'; <rc> = 0
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,796::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::WARNING::2014-04-11 23:00:13,798::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,798::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,832::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,833::sp::329::Storage.StoragePool::(startSpm) Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
>    self._updateDomainsRole()
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
>    domain = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,833::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,834::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,835::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d69c0c5d-efe3-40da-8172-8c5758643b61`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,835::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,835::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,836::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d69c0c5d-efe3-40da-8172-8c5758643b61`::Granted request
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,837::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`9236862c-e98a-44dc-86a1-29e2de279820`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,837::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' for lock type 'exclusive'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,838::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free. Now locking as 'exclusive' (1 active user)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,838::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`9236862c-e98a-44dc-86a1-29e2de279820`::Granted request
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,839::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,839::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' (0 active users)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,839::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free, finding out if anyone is waiting for it.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,840::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb', Clearing records.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,840::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`6447f156-d9eb-4db9-ad58-da3049474424`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,841::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' for lock type 'exclusive'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,841::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free. Now locking as 'exclusive' (1 active user)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,841::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`6447f156-d9eb-4db9-ad58-da3049474424`::Granted request
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,842::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,842::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' (0 active users)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,843::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free, finding out if anyone is waiting for it.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,843::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48', Clearing records.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,844::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,844::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,844::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,845::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,845::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,846::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,847::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,856::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:13,857::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,859::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,860::task::866::TaskManager.Task::(_setError) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/storage/task.py", line 334, in run
>    return self.cmd(*self.argslist, **self.argsdict)
>  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
>    self._updateDomainsRole()
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
>    domain = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,862::task::885::TaskManager.Task::(_run) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Task._run: 9d3d90f5-8732-4a54-854e-573f3fdcd2de () {} failed - stopping task
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,863::task::1211::TaskManager.Task::(stop) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::stopping in state running (force False)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,863::task::990::TaskManager.Task::(_decref) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::ref 1 aborting True
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,863::task::916::TaskManager.Task::(_runJobs) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,864::task::990::TaskManager.Task::(_decref) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::ref 0 aborting True
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,864::task::925::TaskManager.Task::(_doAbort) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Task._doAbort: force False
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,864::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,865::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state running -> state aborting
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,865::task::550::TaskManager.Task::(__state_aborting) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::_aborting: recover policy auto
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,865::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state aborting -> state racquiring
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,866::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state racquiring -> state recovering
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,866::task::798::TaskManager.Task::(_recover) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::_recover
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,866::task::805::TaskManager.Task::(_recover) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::running recovery None
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,867::task::786::TaskManager.Task::(_recoverDone) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Recover Done: state recovering
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,867::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state recovering -> state recovered
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,867::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>}
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,868::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,868::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,868::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,869::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,869::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,870::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
>Thread-29::DEBUG::2014-04-11 23:00:13,991::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 23:00:14,007::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000459157 s, 1.6 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 23:00:14,042::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 23:00:14,053::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000280847 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 23:00:14,368::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:14,369::task::595::TaskManager.Task::(_updateState) Task=`84ce285c-fb07-4c63-abe2-f9958168868e`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:14,369::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:14,369::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:14,370::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}
>Thread-14::INFO::2014-04-11 23:00:14,370::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:14,370::task::1185::TaskManager.Task::(prepare) Task=`84ce285c-fb07-4c63-abe2-f9958168868e`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}}
>Thread-14::DEBUG::2014-04-11 23:00:14,371::task::595::TaskManager.Task::(_updateState) Task=`84ce285c-fb07-4c63-abe2-f9958168868e`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:14,371::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:14,371::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:14,372::task::990::TaskManager.Task::(_decref) Task=`84ce285c-fb07-4c63-abe2-f9958168868e`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:14,382::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:14,383::task::595::TaskManager.Task::(_updateState) Task=`8846179f-ed46-4bb2-95ec-cafb62b1af01`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:14,384::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:14,399::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:14,399::task::1185::TaskManager.Task::(prepare) Task=`8846179f-ed46-4bb2-95ec-cafb62b1af01`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:14,400::task::595::TaskManager.Task::(_updateState) Task=`8846179f-ed46-4bb2-95ec-cafb62b1af01`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:14,400::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:14,400::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:14,401::task::990::TaskManager.Task::(_decref) Task=`8846179f-ed46-4bb2-95ec-cafb62b1af01`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:14,482::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:14,483::task::595::TaskManager.Task::(_updateState) Task=`c2fa8758-45e1-4745-9d24-5d1e1a268fc7`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:14,483::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:14,484::taskManager::161::TaskManager::(clearTask) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de
>Thread-14::DEBUG::2014-04-11 23:00:14,484::taskManager::166::TaskManager::(clearTask) Return.
>Thread-14::INFO::2014-04-11 23:00:14,484::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
>Thread-14::DEBUG::2014-04-11 23:00:14,485::task::1185::TaskManager.Task::(prepare) Task=`c2fa8758-45e1-4745-9d24-5d1e1a268fc7`::finished: None
>Thread-14::DEBUG::2014-04-11 23:00:14,485::task::595::TaskManager.Task::(_updateState) Task=`c2fa8758-45e1-4745-9d24-5d1e1a268fc7`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:14,485::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:14,486::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:14,486::task::990::TaskManager.Task::(_decref) Task=`c2fa8758-45e1-4745-9d24-5d1e1a268fc7`::ref 0 aborting False
>Thread-33::ERROR::2014-04-11 23:00:18,792::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 23:00:18,792::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 23:00:18,793::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-33::DEBUG::2014-04-11 23:00:18,793::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::ERROR::2014-04-11 23:00:18,825::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 23:00:18,826::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 23:00:18,827::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-39::DEBUG::2014-04-11 23:00:18,828::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::DEBUG::2014-04-11 23:00:19,066::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 23:00:19,068::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::ERROR::2014-04-11 23:00:19,110::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 23:00:19,115::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-39::DEBUG::2014-04-11 23:00:19,128::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 23:00:19,134::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::DEBUG::2014-04-11 23:00:19,137::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-39::ERROR::2014-04-11 23:00:19,184::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 23:00:19,185::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-43::DEBUG::2014-04-11 23:00:22,646::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 23:00:22,660::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000500292 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 23:00:23,303::task::595::TaskManager.Task::(_updateState) Task=`de82191d-a9af-4594-a20e-b253eb38a2e0`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:23,303::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 23:00:23,304::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000512307', 'lastCheck': '9.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000322894', 'lastCheck': '9.9', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000459157', 'lastCheck': '9.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000500292', 'lastCheck': '0.6', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000280847', 'lastCheck': '9.2', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:23,305::task::1185::TaskManager.Task::(prepare) Task=`de82191d-a9af-4594-a20e-b253eb38a2e0`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000512307', 'lastCheck': '9.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000322894', 'lastCheck': '9.9', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000459157', 'lastCheck': '9.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000500292', 'lastCheck': '0.6', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000280847', 'lastCheck': '9.2', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:23,305::task::595::TaskManager.Task::(_updateState) Task=`de82191d-a9af-4594-a20e-b253eb38a2e0`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:23,305::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:23,306::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:23,306::task::990::TaskManager.Task::(_decref) Task=`de82191d-a9af-4594-a20e-b253eb38a2e0`::ref 0 aborting False
>Thread-24::DEBUG::2014-04-11 23:00:23,466::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 23:00:23,480::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.00032068 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 23:00:23,638::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 23:00:23,652::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000575685 s, 1.2 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 23:00:24,035::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 23:00:24,048::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000384216 s, 2.0 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 23:00:24,083::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 23:00:24,094::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000271965 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 23:00:24,642::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:24,644::task::595::TaskManager.Task::(_updateState) Task=`10ab6b4e-ce22-44c8-986e-a0d98914e5e2`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:24,644::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:24,659::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:24,659::task::1185::TaskManager.Task::(prepare) Task=`10ab6b4e-ce22-44c8-986e-a0d98914e5e2`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:24,660::task::595::TaskManager.Task::(_updateState) Task=`10ab6b4e-ce22-44c8-986e-a0d98914e5e2`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:24,660::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:24,660::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:24,661::task::990::TaskManager.Task::(_decref) Task=`10ab6b4e-ce22-44c8-986e-a0d98914e5e2`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:24,665::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:24,666::task::595::TaskManager.Task::(_updateState) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:24,666::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-14::ERROR::2014-04-11 23:00:24,667::task::866::TaskManager.Task::(_setError) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2115, in getAllTasksStatuses
>    allTasksStatus = sp.getAllTasksStatuses()
>  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
>    raise SecureError("Secured object is not in safe state")
>SecureError: Secured object is not in safe state
>Thread-14::DEBUG::2014-04-11 23:00:24,667::task::885::TaskManager.Task::(_run) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::Task._run: 4905fe14-b30b-45f2-aee8-fd9523604d46 () {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 23:00:24,668::task::1211::TaskManager.Task::(stop) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 23:00:24,668::task::990::TaskManager.Task::(_decref) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 23:00:24,668::task::1168::TaskManager.Task::(prepare) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
>Thread-14::DEBUG::2014-04-11 23:00:24,668::task::1173::TaskManager.Task::(prepare) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::Prepare: aborted: Secured object is not in safe state
>Thread-14::DEBUG::2014-04-11 23:00:24,669::task::990::TaskManager.Task::(_decref) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 23:00:24,669::task::925::TaskManager.Task::(_doAbort) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 23:00:24,669::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:24,670::task::595::TaskManager.Task::(_updateState) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 23:00:24,670::task::550::TaskManager.Task::(__state_aborting) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 23:00:24,670::task::595::TaskManager.Task::(_updateState) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 23:00:24,670::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:24,671::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 23:00:24,671::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SecureError: Secured object is not in safe state
>Thread-14::DEBUG::2014-04-11 23:00:24,788::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:24,789::task::595::TaskManager.Task::(_updateState) Task=`2f94f4ba-858e-4021-8aa8-e7a5cf0cec2a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:24,789::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:24,802::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:24,802::task::1185::TaskManager.Task::(prepare) Task=`2f94f4ba-858e-4021-8aa8-e7a5cf0cec2a`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:24,802::task::595::TaskManager.Task::(_updateState) Task=`2f94f4ba-858e-4021-8aa8-e7a5cf0cec2a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:24,802::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:24,803::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:24,803::task::990::TaskManager.Task::(_decref) Task=`2f94f4ba-858e-4021-8aa8-e7a5cf0cec2a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:24,850::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:24,850::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:24,851::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
>Thread-14::DEBUG::2014-04-11 23:00:24,852::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`f9634649-0ce2-4e05-8639-a00b6507d0ad`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart'
>Thread-14::DEBUG::2014-04-11 23:00:24,852::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 23:00:24,853::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 23:00:24,853::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`f9634649-0ce2-4e05-8639-a00b6507d0ad`::Granted request
>Thread-14::DEBUG::2014-04-11 23:00:24,854::task::827::TaskManager.Task::(resourceAcquired) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive)
>Thread-14::DEBUG::2014-04-11 23:00:24,854::task::990::TaskManager.Task::(_decref) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::ref 1 aborting False
>Thread-14::INFO::2014-04-11 23:00:24,854::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
>Thread-14::DEBUG::2014-04-11 23:00:24,855::task::1180::TaskManager.Task::(prepare) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Prepare: 1 jobs exist, move to acquiring
>Thread-14::DEBUG::2014-04-11 23:00:24,855::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state preparing -> state acquiring
>Thread-14::DEBUG::2014-04-11 23:00:24,855::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state acquiring -> state queued
>Thread-14::DEBUG::2014-04-11 23:00:24,856::taskManager::50::TaskManager::(_queueTask) queuing task: 0a360c1f-47b6-41f6-a72e-e92a32263e98
>Thread-14::DEBUG::2014-04-11 23:00:24,856::taskManager::56::TaskManager::(_queueTask) task queued: 0a360c1f-47b6-41f6-a72e-e92a32263e98
>9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:24,856::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
>Thread-14::DEBUG::2014-04-11 23:00:24,857::task::1182::TaskManager.Task::(prepare) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::returning
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,857::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 0a360c1f-47b6-41f6-a72e-e92a32263e98 running: <bound method Task.commit of <storage.task.Task instance at 0x2e38878>> with: None
>Thread-14::DEBUG::2014-04-11 23:00:24,858::task::990::TaskManager.Task::(_decref) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::ref 0 aborting False
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,858::task::1194::TaskManager.Task::(commit) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::committing task: 0a360c1f-47b6-41f6-a72e-e92a32263e98
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,859::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state queued -> state running
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,860::task::905::TaskManager.Task::(_runJobs) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {})
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,861::task::331::TaskManager.Task::(run) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,861::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,861::misc::764::SamplingMethod::(__call__) Got in to sampling method
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,878::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,879::misc::772::SamplingMethod::(__call__) Returning last result
>0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:24,892::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,893::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:24,893::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,931::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,931::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,947::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,947::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,948::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,948::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,963::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,964::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`f264ec2c-72a7-4b4b-aaae-d22e90555b03`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,964::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,964::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,965::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`f264ec2c-72a7-4b4b-aaae-d22e90555b03`::Granted request
>0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:24,965::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a`
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,966::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`2580c927-cd8e-44d0-b401-aeaa7122bd01`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,966::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,967::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,967::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`2580c927-cd8e-44d0-b401-aeaa7122bd01`::Granted request
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,983::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,984::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,000::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,002::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:25,002::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered
>0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:25,003::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,003::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,003::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,004::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,004::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,004::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,020::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,020::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
>Thread-390::DEBUG::2014-04-11 23:00:25,021::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-391::DEBUG::2014-04-11 23:00:25,022::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-390::DEBUG::2014-04-11 23:00:25,023::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`9029a627-c541-4cb3-b616-b56015928f9d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-392::DEBUG::2014-04-11 23:00:25,024::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-390::DEBUG::2014-04-11 23:00:25,024::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive'
>Thread-391::DEBUG::2014-04-11 23:00:25,025::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`c8eec928-2658-460b-a41e-7859faf96690`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-393::ERROR::2014-04-11 23:00:25,026::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-392::DEBUG::2014-04-11 23:00:25,026::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`c8efee26-1822-410e-9d26-e32186b6241f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-394::ERROR::2014-04-11 23:00:25,028::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-390::DEBUG::2014-04-11 23:00:25,028::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user)
>Thread-391::DEBUG::2014-04-11 23:00:25,029::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive'
>Thread-395::DEBUG::2014-04-11 23:00:25,029::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 21619c8e-99ea-4813-be02-d708971e5393
>Thread-393::ERROR::2014-04-11 23:00:25,030::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,030::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-392::DEBUG::2014-04-11 23:00:25,031::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive'
>Thread-394::ERROR::2014-04-11 23:00:25,031::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-390::DEBUG::2014-04-11 23:00:25,031::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`9029a627-c541-4cb3-b616-b56015928f9d`::Granted request
>Thread-395::DEBUG::2014-04-11 23:00:25,033::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`a672482e-7fbe-40fd-bcfb-429da3fb499c`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-393::DEBUG::2014-04-11 23:00:25,033::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-391::DEBUG::2014-04-11 23:00:25,035::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user)
>Thread-395::DEBUG::2014-04-11 23:00:25,035::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive'
>Thread-390::DEBUG::2014-04-11 23:00:25,036::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`4bc51ac8-a85d-42a5-b0b8-629458b60845`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-393::DEBUG::2014-04-11 23:00:25,037::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-394::DEBUG::2014-04-11 23:00:25,038::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-391::DEBUG::2014-04-11 23:00:25,038::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`c8eec928-2658-460b-a41e-7859faf96690`::Granted request
>Thread-390::DEBUG::2014-04-11 23:00:25,039::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive'
>Thread-394::DEBUG::2014-04-11 23:00:25,053::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,054::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-391::DEBUG::2014-04-11 23:00:25,057::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`e1eb6b89-c2d4-4754-b833-c0ccbc551bdd`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,057::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-391::DEBUG::2014-04-11 23:00:25,068::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,069::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-392::DEBUG::2014-04-11 23:00:25,071::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user)
>Thread-392::DEBUG::2014-04-11 23:00:25,071::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`c8efee26-1822-410e-9d26-e32186b6241f`::Granted request
>Thread-395::DEBUG::2014-04-11 23:00:25,076::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,077::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms
>Thread-395::DEBUG::2014-04-11 23:00:25,078::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`a672482e-7fbe-40fd-bcfb-429da3fb499c`::Granted request
>Thread-392::DEBUG::2014-04-11 23:00:25,079::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`daa390c2-033f-45ff-9716-a3535fbe9d58`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-390::DEBUG::2014-04-11 23:00:25,080::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,081::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks
>Thread-395::DEBUG::2014-04-11 23:00:25,082::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`5be6d11d-cbb2-421b-a786-4d1628a24236`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-392::DEBUG::2014-04-11 23:00:25,082::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive'
>Thread-390::DEBUG::2014-04-11 23:00:25,083::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`4bc51ac8-a85d-42a5-b0b8-629458b60845`::Granted request
>Thread-395::DEBUG::2014-04-11 23:00:25,084::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive'
>Thread-391::DEBUG::2014-04-11 23:00:25,086::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user)
>Thread-391::DEBUG::2014-04-11 23:00:25,088::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`e1eb6b89-c2d4-4754-b833-c0ccbc551bdd`::Granted request
>Thread-392::DEBUG::2014-04-11 23:00:25,089::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user)
>Thread-391::DEBUG::2014-04-11 23:00:25,090::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` is not a data domain it is an 2 domain, not upgrading
>Thread-392::DEBUG::2014-04-11 23:00:25,090::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`daa390c2-033f-45ff-9716-a3535fbe9d58`::Granted request
>Thread-391::DEBUG::2014-04-11 23:00:25,090::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87'
>Thread-395::DEBUG::2014-04-11 23:00:25,091::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user)
>Thread-395::DEBUG::2014-04-11 23:00:25,092::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`5be6d11d-cbb2-421b-a786-4d1628a24236`::Granted request
>Thread-391::DEBUG::2014-04-11 23:00:25,094::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users)
>Thread-391::DEBUG::2014-04-11 23:00:25,095::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it.
>Thread-391::DEBUG::2014-04-11 23:00:25,096::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records.
>Thread-391::DEBUG::2014-04-11 23:00:25,096::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87'
>Thread-391::DEBUG::2014-04-11 23:00:25,097::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users)
>Thread-391::DEBUG::2014-04-11 23:00:25,097::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it.
>Thread-391::DEBUG::2014-04-11 23:00:25,097::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,101::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>Thread-390::DEBUG::2014-04-11 23:00:25,104::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955']
>Thread-390::DEBUG::2014-04-11 23:00:25,105::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-390::DEBUG::2014-04-11 23:00:25,106::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-390::DEBUG::2014-04-11 23:00:25,106::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-390::DEBUG::2014-04-11 23:00:25,106::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-390::DEBUG::2014-04-11 23:00:25,107::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-390::DEBUG::2014-04-11 23:00:25,107::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-390::DEBUG::2014-04-11 23:00:25,107::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-390::DEBUG::2014-04-11 23:00:25,108::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-392::DEBUG::2014-04-11 23:00:25,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40']
>Thread-392::DEBUG::2014-04-11 23:00:25,110::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce'
>Thread-392::DEBUG::2014-04-11 23:00:25,110::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users)
>Thread-392::DEBUG::2014-04-11 23:00:25,111::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it.
>Thread-392::DEBUG::2014-04-11 23:00:25,111::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records.
>Thread-392::DEBUG::2014-04-11 23:00:25,111::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce'
>Thread-395::DEBUG::2014-04-11 23:00:25,112::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2']
>Thread-392::DEBUG::2014-04-11 23:00:25,113::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users)
>Thread-395::DEBUG::2014-04-11 23:00:25,114::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393'
>Thread-392::DEBUG::2014-04-11 23:00:25,114::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it.
>Thread-392::DEBUG::2014-04-11 23:00:25,115::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records.
>Thread-395::DEBUG::2014-04-11 23:00:25,115::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' (0 active users)
>Thread-395::DEBUG::2014-04-11 23:00:25,116::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it.
>Thread-395::DEBUG::2014-04-11 23:00:25,116::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393', Clearing records.
>Thread-395::DEBUG::2014-04-11 23:00:25,117::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393'
>Thread-395::DEBUG::2014-04-11 23:00:25,117::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' (0 active users)
>Thread-395::DEBUG::2014-04-11 23:00:25,117::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it.
>Thread-395::DEBUG::2014-04-11 23:00:25,118::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393', Clearing records.
>Thread-393::DEBUG::2014-04-11 23:00:25,320::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-393::WARNING::2014-04-11 23:00:25,322::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-394::DEBUG::2014-04-11 23:00:25,358::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-393::ERROR::2014-04-11 23:00:25,359::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,363::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,367::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-394::WARNING::2014-04-11 23:00:25,371::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,373::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,399::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-394::ERROR::2014-04-11 23:00:25,429::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,679::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>0a360c1f-47b6-41f6-a72e-e92a32263e98::WARNING::2014-04-11 23:00:25,681::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,681::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,714::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,714::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
>    self.checkBackupDomain(__securityOverride=True)
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
>    dom = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,716::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,717::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,717::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,718::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:00:25,866::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:25,868::task::595::TaskManager.Task::(_updateState) Task=`31adae1f-0705-498a-805e-436ecda55a34`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:25,868::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='0a360c1f-47b6-41f6-a72e-e92a32263e98', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:25,868::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 0a360c1f-47b6-41f6-a72e-e92a32263e98
>Thread-14::DEBUG::2014-04-11 23:00:25,869::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}
>Thread-14::INFO::2014-04-11 23:00:25,869::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}}
>Thread-14::DEBUG::2014-04-11 23:00:25,870::task::1185::TaskManager.Task::(prepare) Task=`31adae1f-0705-498a-805e-436ecda55a34`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}}
>Thread-14::DEBUG::2014-04-11 23:00:25,870::task::595::TaskManager.Task::(_updateState) Task=`31adae1f-0705-498a-805e-436ecda55a34`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:25,870::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:25,871::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:25,871::task::990::TaskManager.Task::(_decref) Task=`31adae1f-0705-498a-805e-436ecda55a34`::ref 0 aborting False
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,986::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>0a360c1f-47b6-41f6-a72e-e92a32263e98::WARNING::2014-04-11 23:00:26,000::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,000::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:26,035::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:26,036::sp::329::Storage.StoragePool::(startSpm) Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
>    self._updateDomainsRole()
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
>    domain = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:26,037::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,037::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,038::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`545fbe48-1e92-47b2-b3e9-4090fed5bb56`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,038::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,039::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,039::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`545fbe48-1e92-47b2-b3e9-4090fed5bb56`::Granted request
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,040::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`c55ebd22-d893-4f3f-8b7b-90a5139d7ac3`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,040::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' for lock type 'exclusive'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,041::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free. Now locking as 'exclusive' (1 active user)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,041::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`c55ebd22-d893-4f3f-8b7b-90a5139d7ac3`::Granted request
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,042::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,042::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' (0 active users)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,043::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free, finding out if anyone is waiting for it.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,043::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb', Clearing records.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,044::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`192c4567-9ad1-4b2f-bdac-b86dc3b60438`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,044::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' for lock type 'exclusive'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,044::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free. Now locking as 'exclusive' (1 active user)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,045::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`192c4567-9ad1-4b2f-bdac-b86dc3b60438`::Granted request
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,045::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,046::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' (0 active users)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,046::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free, finding out if anyone is waiting for it.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,046::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48', Clearing records.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,047::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,047::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,048::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,048::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,049::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,050::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,050::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,059::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:26,059::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,062::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released
>0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:26,063::task::866::TaskManager.Task::(_setError) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/storage/task.py", line 334, in run
>    return self.cmd(*self.argslist, **self.argsdict)
>  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
>    self._updateDomainsRole()
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
>    domain = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,064::task::885::TaskManager.Task::(_run) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Task._run: 0a360c1f-47b6-41f6-a72e-e92a32263e98 () {} failed - stopping task
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,064::task::1211::TaskManager.Task::(stop) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::stopping in state running (force False)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,065::task::990::TaskManager.Task::(_decref) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::ref 1 aborting True
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,065::task::916::TaskManager.Task::(_runJobs) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,065::task::990::TaskManager.Task::(_decref) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::ref 0 aborting True
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,066::task::925::TaskManager.Task::(_doAbort) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Task._doAbort: force False
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,066::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,066::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state running -> state aborting
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,067::task::550::TaskManager.Task::(__state_aborting) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::_aborting: recover policy auto
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,067::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state aborting -> state racquiring
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,067::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state racquiring -> state recovering
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,068::task::798::TaskManager.Task::(_recover) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::_recover
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,068::task::805::TaskManager.Task::(_recover) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::running recovery None
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,068::task::786::TaskManager.Task::(_recoverDone) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Recover Done: state recovering
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,069::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state recovering -> state recovered
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,069::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>}
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,069::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,070::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,070::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,070::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,071::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,071::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
>Thread-14::DEBUG::2014-04-11 23:00:26,876::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:26,877::task::595::TaskManager.Task::(_updateState) Task=`50293260-22a4-4c28-8b44-487ae2f5fcca`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:26,878::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='0a360c1f-47b6-41f6-a72e-e92a32263e98', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:26,878::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 0a360c1f-47b6-41f6-a72e-e92a32263e98
>Thread-14::DEBUG::2014-04-11 23:00:26,878::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}
>Thread-14::INFO::2014-04-11 23:00:26,879::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}}
>Thread-14::DEBUG::2014-04-11 23:00:26,879::task::1185::TaskManager.Task::(prepare) Task=`50293260-22a4-4c28-8b44-487ae2f5fcca`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}}
>Thread-14::DEBUG::2014-04-11 23:00:26,879::task::595::TaskManager.Task::(_updateState) Task=`50293260-22a4-4c28-8b44-487ae2f5fcca`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:26,880::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:26,880::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:26,880::task::990::TaskManager.Task::(_decref) Task=`50293260-22a4-4c28-8b44-487ae2f5fcca`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:26,894::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:26,899::task::595::TaskManager.Task::(_updateState) Task=`d20f7218-ba2a-4a82-99bb-0c17cfcf6ca8`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:26,902::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:26,920::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:26,923::task::1185::TaskManager.Task::(prepare) Task=`d20f7218-ba2a-4a82-99bb-0c17cfcf6ca8`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:26,925::task::595::TaskManager.Task::(_updateState) Task=`d20f7218-ba2a-4a82-99bb-0c17cfcf6ca8`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:26,928::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:26,931::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:26,934::task::990::TaskManager.Task::(_decref) Task=`d20f7218-ba2a-4a82-99bb-0c17cfcf6ca8`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:27,042::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:27,043::task::595::TaskManager.Task::(_updateState) Task=`1fec0d62-3d9e-4ef4-a9ca-92f9ac4a9e5a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:27,043::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='0a360c1f-47b6-41f6-a72e-e92a32263e98', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:27,044::taskManager::161::TaskManager::(clearTask) Entry. taskID: 0a360c1f-47b6-41f6-a72e-e92a32263e98
>Thread-14::DEBUG::2014-04-11 23:00:27,044::taskManager::166::TaskManager::(clearTask) Return.
>Thread-14::INFO::2014-04-11 23:00:27,044::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
>Thread-14::DEBUG::2014-04-11 23:00:27,045::task::1185::TaskManager.Task::(prepare) Task=`1fec0d62-3d9e-4ef4-a9ca-92f9ac4a9e5a`::finished: None
>Thread-14::DEBUG::2014-04-11 23:00:27,045::task::595::TaskManager.Task::(_updateState) Task=`1fec0d62-3d9e-4ef4-a9ca-92f9ac4a9e5a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:27,045::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:27,046::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:27,046::task::990::TaskManager.Task::(_decref) Task=`1fec0d62-3d9e-4ef4-a9ca-92f9ac4a9e5a`::ref 0 aborting False
>Thread-33::ERROR::2014-04-11 23:00:29,125::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 23:00:29,125::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 23:00:29,126::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-33::DEBUG::2014-04-11 23:00:29,127::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-39::ERROR::2014-04-11 23:00:29,186::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 23:00:29,187::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 23:00:29,188::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-39::DEBUG::2014-04-11 23:00:29,190::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-33::DEBUG::2014-04-11 23:00:29,397::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 23:00:29,399::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::ERROR::2014-04-11 23:00:29,431::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 23:00:29,432::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-39::DEBUG::2014-04-11 23:00:29,465::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 23:00:29,467::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::DEBUG::2014-04-11 23:00:29,467::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-39::ERROR::2014-04-11 23:00:29,500::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 23:00:29,500::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-43::DEBUG::2014-04-11 23:00:32,686::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 23:00:32,700::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000495644 s, 1.5 MB/s\n'; <rc> = 0
>Thread-24::DEBUG::2014-04-11 23:00:33,509::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 23:00:33,522::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000393856 s, 937 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 23:00:33,678::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 23:00:33,691::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000395372 s, 1.8 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 23:00:34,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 23:00:34,098::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000358786 s, 2.1 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 23:00:34,130::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 23:00:34,142::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000328121 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 23:00:37,206::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:37,245::task::595::TaskManager.Task::(_updateState) Task=`1ce9ac53-3258-432b-a837-283dd83d34dd`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:37,245::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:37,261::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:37,261::task::1185::TaskManager.Task::(prepare) Task=`1ce9ac53-3258-432b-a837-283dd83d34dd`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:37,262::task::595::TaskManager.Task::(_updateState) Task=`1ce9ac53-3258-432b-a837-283dd83d34dd`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:37,262::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:37,262::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:37,263::task::990::TaskManager.Task::(_decref) Task=`1ce9ac53-3258-432b-a837-283dd83d34dd`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:37,267::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:37,268::task::595::TaskManager.Task::(_updateState) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:37,268::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-14::ERROR::2014-04-11 23:00:37,268::task::866::TaskManager.Task::(_setError) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2115, in getAllTasksStatuses
>    allTasksStatus = sp.getAllTasksStatuses()
>  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
>    raise SecureError("Secured object is not in safe state")
>SecureError: Secured object is not in safe state
>Thread-14::DEBUG::2014-04-11 23:00:37,269::task::885::TaskManager.Task::(_run) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::Task._run: 05cf4d7b-f676-47ac-8313-8941fca0ab57 () {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 23:00:37,269::task::1211::TaskManager.Task::(stop) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 23:00:37,270::task::990::TaskManager.Task::(_decref) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 23:00:37,270::task::1168::TaskManager.Task::(prepare) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
>Thread-14::DEBUG::2014-04-11 23:00:37,270::task::1173::TaskManager.Task::(prepare) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::Prepare: aborted: Secured object is not in safe state
>Thread-14::DEBUG::2014-04-11 23:00:37,271::task::990::TaskManager.Task::(_decref) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 23:00:37,271::task::925::TaskManager.Task::(_doAbort) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 23:00:37,271::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:37,271::task::595::TaskManager.Task::(_updateState) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 23:00:37,272::task::550::TaskManager.Task::(__state_aborting) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 23:00:37,272::task::595::TaskManager.Task::(_updateState) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 23:00:37,272::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:37,272::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 23:00:37,273::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SecureError: Secured object is not in safe state
>Thread-14::DEBUG::2014-04-11 23:00:37,392::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:37,393::task::595::TaskManager.Task::(_updateState) Task=`f85ca92a-8f57-4e28-854d-282e00bb8308`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:37,393::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:37,406::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:37,407::task::1185::TaskManager.Task::(prepare) Task=`f85ca92a-8f57-4e28-854d-282e00bb8308`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:37,407::task::595::TaskManager.Task::(_updateState) Task=`f85ca92a-8f57-4e28-854d-282e00bb8308`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:37,407::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:37,407::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:37,408::task::990::TaskManager.Task::(_decref) Task=`f85ca92a-8f57-4e28-854d-282e00bb8308`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:37,456::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:37,457::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:37,457::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
>Thread-14::DEBUG::2014-04-11 23:00:37,458::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`1e904999-00f0-4c11-9fbc-28e0a09e92d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart'
>Thread-14::DEBUG::2014-04-11 23:00:37,459::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 23:00:37,459::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 23:00:37,459::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`1e904999-00f0-4c11-9fbc-28e0a09e92d2`::Granted request
>Thread-14::DEBUG::2014-04-11 23:00:37,460::task::827::TaskManager.Task::(resourceAcquired) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive)
>Thread-14::DEBUG::2014-04-11 23:00:37,460::task::990::TaskManager.Task::(_decref) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::ref 1 aborting False
>Thread-14::INFO::2014-04-11 23:00:37,461::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
>Thread-14::DEBUG::2014-04-11 23:00:37,461::task::1180::TaskManager.Task::(prepare) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Prepare: 1 jobs exist, move to acquiring
>Thread-14::DEBUG::2014-04-11 23:00:37,461::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state preparing -> state acquiring
>Thread-14::DEBUG::2014-04-11 23:00:37,462::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state acquiring -> state queued
>Thread-14::DEBUG::2014-04-11 23:00:37,462::taskManager::50::TaskManager::(_queueTask) queuing task: 9ceac723-a09b-426b-9729-c3d1775660a9
>Thread-14::DEBUG::2014-04-11 23:00:37,462::taskManager::56::TaskManager::(_queueTask) task queued: 9ceac723-a09b-426b-9729-c3d1775660a9
>Thread-10::DEBUG::2014-04-11 23:00:37,463::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
>Thread-14::DEBUG::2014-04-11 23:00:37,463::task::1182::TaskManager.Task::(prepare) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::returning
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,463::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 9ceac723-a09b-426b-9729-c3d1775660a9 running: <bound method Task.commit of <storage.task.Task instance at 0x2e38878>> with: None
>Thread-14::DEBUG::2014-04-11 23:00:37,464::task::990::TaskManager.Task::(_decref) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::ref 0 aborting False
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,465::task::1194::TaskManager.Task::(commit) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::committing task: 9ceac723-a09b-426b-9729-c3d1775660a9
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,466::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state queued -> state running
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,466::task::905::TaskManager.Task::(_runJobs) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {})
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,467::task::331::TaskManager.Task::(run) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,468::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,468::misc::764::SamplingMethod::(__call__) Got in to sampling method
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,486::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,486::misc::772::SamplingMethod::(__call__) Returning last result
>9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,499::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,500::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,500::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,543::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,543::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,559::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,559::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,560::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,560::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,583::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,584::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`74f47895-90e6-4be0-9bb1-a37fe6924576`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,584::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,585::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,585::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`74f47895-90e6-4be0-9bb1-a37fe6924576`::Granted request
>9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,586::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a`
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,586::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`4df9b8aa-031b-4b60-bc6b-186a827cda7d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,587::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,587::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,587::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`4df9b8aa-031b-4b60-bc6b-186a827cda7d`::Granted request
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,602::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,604::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,620::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,622::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,622::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered
>9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,623::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,623::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,623::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,624::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,624::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,624::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,640::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,640::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,641::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
>Thread-438::DEBUG::2014-04-11 23:00:37,641::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-438::DEBUG::2014-04-11 23:00:37,643::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`552f1acb-2df2-4ca5-82dc-4eeb71313107`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-439::DEBUG::2014-04-11 23:00:37,643::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-439::DEBUG::2014-04-11 23:00:37,645::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`92c2227e-8e51-4308-8d5d-38a6a1bfed5e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-438::DEBUG::2014-04-11 23:00:37,645::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive'
>Thread-440::DEBUG::2014-04-11 23:00:37,645::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-439::DEBUG::2014-04-11 23:00:37,647::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive'
>Thread-438::DEBUG::2014-04-11 23:00:37,647::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user)
>Thread-441::ERROR::2014-04-11 23:00:37,647::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-442::ERROR::2014-04-11 23:00:37,649::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-440::DEBUG::2014-04-11 23:00:37,651::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`01d8ffab-8ab4-4066-a37b-6008760bdd06`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-438::DEBUG::2014-04-11 23:00:37,652::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`552f1acb-2df2-4ca5-82dc-4eeb71313107`::Granted request
>Thread-443::DEBUG::2014-04-11 23:00:37,652::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 21619c8e-99ea-4813-be02-d708971e5393
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,652::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-441::ERROR::2014-04-11 23:00:37,653::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-442::ERROR::2014-04-11 23:00:37,653::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-440::DEBUG::2014-04-11 23:00:37,653::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive'
>Thread-443::DEBUG::2014-04-11 23:00:37,655::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`31a554ec-565a-4d91-bef3-01e993fc2155`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-439::DEBUG::2014-04-11 23:00:37,655::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user)
>Thread-441::DEBUG::2014-04-11 23:00:37,656::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-438::DEBUG::2014-04-11 23:00:37,657::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`e26de70a-5033-4dac-a9be-7b4f73b8ee3f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-443::DEBUG::2014-04-11 23:00:37,658::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive'
>Thread-439::DEBUG::2014-04-11 23:00:37,658::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`92c2227e-8e51-4308-8d5d-38a6a1bfed5e`::Granted request
>Thread-442::DEBUG::2014-04-11 23:00:37,659::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-441::DEBUG::2014-04-11 23:00:37,660::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-438::DEBUG::2014-04-11 23:00:37,660::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,661::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-442::DEBUG::2014-04-11 23:00:37,662::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-439::DEBUG::2014-04-11 23:00:37,663::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`dcf38eae-4367-4806-95c8-6b387fef2fb1`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,676::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-439::DEBUG::2014-04-11 23:00:37,687::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,688::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-440::DEBUG::2014-04-11 23:00:37,690::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user)
>Thread-440::DEBUG::2014-04-11 23:00:37,691::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`01d8ffab-8ab4-4066-a37b-6008760bdd06`::Granted request
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,692::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms
>Thread-443::DEBUG::2014-04-11 23:00:37,694::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user)
>Thread-440::DEBUG::2014-04-11 23:00:37,695::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`e75021d1-c0a3-4edd-a594-da6ed73d96ac`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-443::DEBUG::2014-04-11 23:00:37,695::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`31a554ec-565a-4d91-bef3-01e993fc2155`::Granted request
>Thread-440::DEBUG::2014-04-11 23:00:37,696::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,696::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks
>Thread-438::DEBUG::2014-04-11 23:00:37,697::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user)
>Thread-438::DEBUG::2014-04-11 23:00:37,700::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`e26de70a-5033-4dac-a9be-7b4f73b8ee3f`::Granted request
>Thread-443::DEBUG::2014-04-11 23:00:37,701::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`cccb83fe-4a9f-4f71-b5a6-78c659ad1668`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-443::DEBUG::2014-04-11 23:00:37,702::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive'
>Thread-439::DEBUG::2014-04-11 23:00:37,703::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user)
>Thread-439::DEBUG::2014-04-11 23:00:37,704::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`dcf38eae-4367-4806-95c8-6b387fef2fb1`::Granted request
>Thread-439::DEBUG::2014-04-11 23:00:37,705::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` is not a data domain it is an 2 domain, not upgrading
>Thread-440::DEBUG::2014-04-11 23:00:37,705::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user)
>Thread-439::DEBUG::2014-04-11 23:00:37,706::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87'
>Thread-440::DEBUG::2014-04-11 23:00:37,706::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`e75021d1-c0a3-4edd-a594-da6ed73d96ac`::Granted request
>Thread-443::DEBUG::2014-04-11 23:00:37,708::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user)
>Thread-443::DEBUG::2014-04-11 23:00:37,709::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`cccb83fe-4a9f-4f71-b5a6-78c659ad1668`::Granted request
>Thread-439::DEBUG::2014-04-11 23:00:37,711::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users)
>Thread-439::DEBUG::2014-04-11 23:00:37,711::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it.
>Thread-439::DEBUG::2014-04-11 23:00:37,711::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records.
>Thread-439::DEBUG::2014-04-11 23:00:37,712::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87'
>Thread-439::DEBUG::2014-04-11 23:00:37,712::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users)
>Thread-439::DEBUG::2014-04-11 23:00:37,713::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it.
>Thread-439::DEBUG::2014-04-11 23:00:37,713::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,717::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>Thread-438::DEBUG::2014-04-11 23:00:37,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955']
>Thread-438::DEBUG::2014-04-11 23:00:37,722::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-438::DEBUG::2014-04-11 23:00:37,723::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-438::DEBUG::2014-04-11 23:00:37,723::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-438::DEBUG::2014-04-11 23:00:37,723::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-438::DEBUG::2014-04-11 23:00:37,724::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-438::DEBUG::2014-04-11 23:00:37,724::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-438::DEBUG::2014-04-11 23:00:37,724::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-438::DEBUG::2014-04-11 23:00:37,725::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-440::DEBUG::2014-04-11 23:00:37,726::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40']
>Thread-440::DEBUG::2014-04-11 23:00:37,727::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce'
>Thread-440::DEBUG::2014-04-11 23:00:37,727::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users)
>Thread-440::DEBUG::2014-04-11 23:00:37,727::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it.
>Thread-440::DEBUG::2014-04-11 23:00:37,728::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records.
>Thread-440::DEBUG::2014-04-11 23:00:37,728::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce'
>Thread-440::DEBUG::2014-04-11 23:00:37,728::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users)
>Thread-440::DEBUG::2014-04-11 23:00:37,729::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it.
>Thread-440::DEBUG::2014-04-11 23:00:37,729::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records.
>Thread-443::DEBUG::2014-04-11 23:00:37,730::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2']
>Thread-443::DEBUG::2014-04-11 23:00:37,731::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393'
>Thread-443::DEBUG::2014-04-11 23:00:37,731::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' (0 active users)
>Thread-443::DEBUG::2014-04-11 23:00:37,732::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it.
>Thread-443::DEBUG::2014-04-11 23:00:37,732::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393', Clearing records.
>Thread-443::DEBUG::2014-04-11 23:00:37,732::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393'
>Thread-443::DEBUG::2014-04-11 23:00:37,733::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' (0 active users)
>Thread-443::DEBUG::2014-04-11 23:00:37,733::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it.
>Thread-443::DEBUG::2014-04-11 23:00:37,733::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393', Clearing records.
>Thread-441::DEBUG::2014-04-11 23:00:37,942::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-441::WARNING::2014-04-11 23:00:37,944::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-442::DEBUG::2014-04-11 23:00:37,983::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-441::ERROR::2014-04-11 23:00:37,983::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:37,990::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:37,995::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-442::WARNING::2014-04-11 23:00:37,995::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,999::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,029::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-442::ERROR::2014-04-11 23:00:38,059::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,309::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>9ceac723-a09b-426b-9729-c3d1775660a9::WARNING::2014-04-11 23:00:38,311::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,311::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,343::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,344::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
>    self.checkBackupDomain(__securityOverride=True)
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
>    dom = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,346::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,346::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,346::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,347::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:00:38,473::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:38,475::task::595::TaskManager.Task::(_updateState) Task=`4895e8fc-c1f1-41c1-ba22-283b27b6e991`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:38,475::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9ceac723-a09b-426b-9729-c3d1775660a9', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:38,476::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9ceac723-a09b-426b-9729-c3d1775660a9
>Thread-14::DEBUG::2014-04-11 23:00:38,476::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}
>Thread-14::INFO::2014-04-11 23:00:38,476::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}}
>Thread-14::DEBUG::2014-04-11 23:00:38,477::task::1185::TaskManager.Task::(prepare) Task=`4895e8fc-c1f1-41c1-ba22-283b27b6e991`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}}
>Thread-14::DEBUG::2014-04-11 23:00:38,477::task::595::TaskManager.Task::(_updateState) Task=`4895e8fc-c1f1-41c1-ba22-283b27b6e991`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:38,477::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:38,478::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:38,478::task::990::TaskManager.Task::(_decref) Task=`4895e8fc-c1f1-41c1-ba22-283b27b6e991`::ref 0 aborting False
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,615::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>9ceac723-a09b-426b-9729-c3d1775660a9::WARNING::2014-04-11 23:00:38,617::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,617::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,655::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,658::sp::329::Storage.StoragePool::(startSpm) Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
>    self._updateDomainsRole()
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
>    domain = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 23:00:38,661::task::595::TaskManager.Task::(_updateState) Task=`e9064fc3-7f9d-4a81-8cf0-f9a7677ab3b6`::moving from state init -> state preparing
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,662::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,664::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
>Thread-14::INFO::2014-04-11 23:00:38,664::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,668::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`8629862c-6c9e-438d-8455-c27a6a9d082a`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,670::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::INFO::2014-04-11 23:00:38,670::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000395372', 'lastCheck': '5.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393856', 'lastCheck': '5.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000358786', 'lastCheck': '4.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000495644', 'lastCheck': '6.0', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000328121', 'lastCheck': '4.5', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:38,672::task::1185::TaskManager.Task::(prepare) Task=`e9064fc3-7f9d-4a81-8cf0-f9a7677ab3b6`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000395372', 'lastCheck': '5.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393856', 'lastCheck': '5.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000358786', 'lastCheck': '4.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000495644', 'lastCheck': '6.0', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000328121', 'lastCheck': '4.5', 'valid': True}}
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,673::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 23:00:38,674::task::595::TaskManager.Task::(_updateState) Task=`e9064fc3-7f9d-4a81-8cf0-f9a7677ab3b6`::moving from state preparing -> state finished
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,675::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`8629862c-6c9e-438d-8455-c27a6a9d082a`::Granted request
>Thread-14::DEBUG::2014-04-11 23:00:38,676::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:38,678::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,681::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`e717d49d-f713-4fe4-a096-b03ed289c70e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
>Thread-14::DEBUG::2014-04-11 23:00:38,681::task::990::TaskManager.Task::(_decref) Task=`e9064fc3-7f9d-4a81-8cf0-f9a7677ab3b6`::ref 0 aborting False
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,682::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' for lock type 'exclusive'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,685::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free. Now locking as 'exclusive' (1 active user)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,688::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`e717d49d-f713-4fe4-a096-b03ed289c70e`::Granted request
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,693::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,696::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' (0 active users)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,698::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free, finding out if anyone is waiting for it.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,700::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb', Clearing records.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,705::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`b8633a53-53cd-4982-bb93-c2efd85bb46f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,707::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' for lock type 'exclusive'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,710::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free. Now locking as 'exclusive' (1 active user)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,712::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`b8633a53-53cd-4982-bb93-c2efd85bb46f`::Granted request
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,716::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,719::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' (0 active users)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,721::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free, finding out if anyone is waiting for it.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,723::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48', Clearing records.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,728::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,730::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,731::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,732::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,733::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,733::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,734::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,743::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:38,743::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,746::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released
>9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,746::task::866::TaskManager.Task::(_setError) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/storage/task.py", line 334, in run
>    return self.cmd(*self.argslist, **self.argsdict)
>  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
>    self._updateDomainsRole()
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
>    domain = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,747::task::885::TaskManager.Task::(_run) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Task._run: 9ceac723-a09b-426b-9729-c3d1775660a9 () {} failed - stopping task
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,748::task::1211::TaskManager.Task::(stop) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::stopping in state running (force False)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,749::task::990::TaskManager.Task::(_decref) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::ref 1 aborting True
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,751::task::916::TaskManager.Task::(_runJobs) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,753::task::990::TaskManager.Task::(_decref) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::ref 0 aborting True
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,755::task::925::TaskManager.Task::(_doAbort) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Task._doAbort: force False
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,756::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,756::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state running -> state aborting
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,756::task::550::TaskManager.Task::(__state_aborting) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::_aborting: recover policy auto
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,757::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state aborting -> state racquiring
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,757::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state racquiring -> state recovering
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,757::task::798::TaskManager.Task::(_recover) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::_recover
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,758::task::805::TaskManager.Task::(_recover) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::running recovery None
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,758::task::786::TaskManager.Task::(_recoverDone) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Recover Done: state recovering
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,758::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state recovering -> state recovered
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,758::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>}
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,759::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,759::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,760::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,760::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,760::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,761::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
>Thread-33::ERROR::2014-04-11 23:00:39,433::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 23:00:39,434::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 23:00:39,434::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-33::DEBUG::2014-04-11 23:00:39,436::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:00:39,485::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:39,486::task::595::TaskManager.Task::(_updateState) Task=`9c0a6294-2b54-4799-8114-f6619ad69b58`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:39,486::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9ceac723-a09b-426b-9729-c3d1775660a9', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:39,487::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9ceac723-a09b-426b-9729-c3d1775660a9
>Thread-14::DEBUG::2014-04-11 23:00:39,487::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}
>Thread-14::INFO::2014-04-11 23:00:39,488::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}}
>Thread-14::DEBUG::2014-04-11 23:00:39,488::task::1185::TaskManager.Task::(prepare) Task=`9c0a6294-2b54-4799-8114-f6619ad69b58`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}}
>Thread-14::DEBUG::2014-04-11 23:00:39,488::task::595::TaskManager.Task::(_updateState) Task=`9c0a6294-2b54-4799-8114-f6619ad69b58`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:39,489::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:39,489::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:39,489::task::990::TaskManager.Task::(_decref) Task=`9c0a6294-2b54-4799-8114-f6619ad69b58`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:39,500::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:39,501::task::595::TaskManager.Task::(_updateState) Task=`32f221a2-8e14-4eaf-b713-6c0afdea02d2`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:39,502::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-39::ERROR::2014-04-11 23:00:39,502::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 23:00:39,505::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 23:00:39,505::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-39::DEBUG::2014-04-11 23:00:39,507::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-14::INFO::2014-04-11 23:00:39,530::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:39,531::task::1185::TaskManager.Task::(prepare) Task=`32f221a2-8e14-4eaf-b713-6c0afdea02d2`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:39,532::task::595::TaskManager.Task::(_updateState) Task=`32f221a2-8e14-4eaf-b713-6c0afdea02d2`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:39,533::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:39,533::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:39,534::task::990::TaskManager.Task::(_decref) Task=`32f221a2-8e14-4eaf-b713-6c0afdea02d2`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:39,616::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-14::DEBUG::2014-04-11 23:00:39,618::task::595::TaskManager.Task::(_updateState) Task=`18e6c9c0-1975-4001-b38a-0f81fd194d2c`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:39,618::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='9ceac723-a09b-426b-9729-c3d1775660a9', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:39,619::taskManager::161::TaskManager::(clearTask) Entry. taskID: 9ceac723-a09b-426b-9729-c3d1775660a9
>Thread-14::DEBUG::2014-04-11 23:00:39,619::taskManager::166::TaskManager::(clearTask) Return.
>Thread-14::INFO::2014-04-11 23:00:39,620::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
>Thread-14::DEBUG::2014-04-11 23:00:39,621::task::1185::TaskManager.Task::(prepare) Task=`18e6c9c0-1975-4001-b38a-0f81fd194d2c`::finished: None
>Thread-14::DEBUG::2014-04-11 23:00:39,621::task::595::TaskManager.Task::(_updateState) Task=`18e6c9c0-1975-4001-b38a-0f81fd194d2c`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:39,622::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:39,622::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:39,623::task::990::TaskManager.Task::(_decref) Task=`18e6c9c0-1975-4001-b38a-0f81fd194d2c`::ref 0 aborting False
>Thread-33::DEBUG::2014-04-11 23:00:39,708::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 23:00:39,710::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-33::ERROR::2014-04-11 23:00:39,743::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-33::ERROR::2014-04-11 23:00:39,744::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-39::DEBUG::2014-04-11 23:00:39,790::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 23:00:39,801::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::DEBUG::2014-04-11 23:00:39,802::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-39::ERROR::2014-04-11 23:00:39,842::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 23:00:39,842::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-43::DEBUG::2014-04-11 23:00:42,725::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 23:00:42,739::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000453198 s, 1.7 MB/s\n'; <rc> = 0
>Thread-24::DEBUG::2014-04-11 23:00:43,550::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 23:00:43,564::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000362865 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 23:00:43,720::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 23:00:43,734::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000599656 s, 1.2 MB/s\n'; <rc> = 0
>Thread-29::DEBUG::2014-04-11 23:00:44,131::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 23:00:44,143::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000564156 s, 1.3 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 23:00:44,174::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 23:00:44,185::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000302579 s, 2.5 MB/s\n'; <rc> = 0
>Thread-33::ERROR::2014-04-11 23:00:49,745::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::ERROR::2014-04-11 23:00:49,746::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 23:00:49,747::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-33::DEBUG::2014-04-11 23:00:49,748::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:00:49,799::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61]
>Thread-14::DEBUG::2014-04-11 23:00:49,800::task::595::TaskManager.Task::(_updateState) Task=`f3870501-e6df-4d6f-a67a-8b12780d422a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:49,801::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:49,816::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:49,817::task::1185::TaskManager.Task::(prepare) Task=`f3870501-e6df-4d6f-a67a-8b12780d422a`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:49,817::task::595::TaskManager.Task::(_updateState) Task=`f3870501-e6df-4d6f-a67a-8b12780d422a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:49,817::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:49,818::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:49,818::task::990::TaskManager.Task::(_decref) Task=`f3870501-e6df-4d6f-a67a-8b12780d422a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:49,823::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61]
>Thread-14::DEBUG::2014-04-11 23:00:49,823::task::595::TaskManager.Task::(_updateState) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:49,824::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-14::ERROR::2014-04-11 23:00:49,824::task::866::TaskManager.Task::(_setError) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2115, in getAllTasksStatuses
>    allTasksStatus = sp.getAllTasksStatuses()
>  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
>    raise SecureError("Secured object is not in safe state")
>SecureError: Secured object is not in safe state
>Thread-14::DEBUG::2014-04-11 23:00:49,825::task::885::TaskManager.Task::(_run) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::Task._run: e987ef15-ed21-40b3-9b34-c2489cd0c106 () {} failed - stopping task
>Thread-14::DEBUG::2014-04-11 23:00:49,825::task::1211::TaskManager.Task::(stop) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::stopping in state preparing (force False)
>Thread-14::DEBUG::2014-04-11 23:00:49,826::task::990::TaskManager.Task::(_decref) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::ref 1 aborting True
>Thread-14::INFO::2014-04-11 23:00:49,826::task::1168::TaskManager.Task::(prepare) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
>Thread-14::DEBUG::2014-04-11 23:00:49,826::task::1173::TaskManager.Task::(prepare) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::Prepare: aborted: Secured object is not in safe state
>Thread-14::DEBUG::2014-04-11 23:00:49,827::task::990::TaskManager.Task::(_decref) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::ref 0 aborting True
>Thread-14::DEBUG::2014-04-11 23:00:49,827::task::925::TaskManager.Task::(_doAbort) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::Task._doAbort: force False
>Thread-14::DEBUG::2014-04-11 23:00:49,827::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:49,828::task::595::TaskManager.Task::(_updateState) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::moving from state preparing -> state aborting
>Thread-14::DEBUG::2014-04-11 23:00:49,828::task::550::TaskManager.Task::(__state_aborting) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::_aborting: recover policy none
>Thread-14::DEBUG::2014-04-11 23:00:49,828::task::595::TaskManager.Task::(_updateState) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::moving from state aborting -> state failed
>Thread-14::DEBUG::2014-04-11 23:00:49,829::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:49,829::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::ERROR::2014-04-11 23:00:49,829::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SecureError: Secured object is not in safe state
>Thread-39::ERROR::2014-04-11 23:00:49,843::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 23:00:49,844::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 23:00:49,844::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-39::DEBUG::2014-04-11 23:00:49,845::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:00:49,953::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61]
>Thread-14::DEBUG::2014-04-11 23:00:49,955::task::595::TaskManager.Task::(_updateState) Task=`164fe45f-522d-450e-8b83-9968c6c4241d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:49,956::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:49,970::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:49,970::task::1185::TaskManager.Task::(prepare) Task=`164fe45f-522d-450e-8b83-9968c6c4241d`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:49,971::task::595::TaskManager.Task::(_updateState) Task=`164fe45f-522d-450e-8b83-9968c6c4241d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:49,971::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:49,972::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:49,973::task::990::TaskManager.Task::(_decref) Task=`164fe45f-522d-450e-8b83-9968c6c4241d`::ref 0 aborting False
>Thread-33::DEBUG::2014-04-11 23:00:50,021::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-33::WARNING::2014-04-11 23:00:50,024::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-14::DEBUG::2014-04-11 23:00:50,050::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61]
>Thread-14::DEBUG::2014-04-11 23:00:50,062::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state init -> state preparing
>Thread-33::ERROR::2014-04-11 23:00:50,065::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::INFO::2014-04-11 23:00:50,068::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
>Thread-33::ERROR::2014-04-11 23:00:50,070::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>Thread-14::DEBUG::2014-04-11 23:00:50,076::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`59d35f2b-7bbc-490d-ad3f-b3a4ba831fe3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart'
>Thread-14::DEBUG::2014-04-11 23:00:50,077::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 23:00:50,077::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 23:00:50,078::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`59d35f2b-7bbc-490d-ad3f-b3a4ba831fe3`::Granted request
>Thread-14::DEBUG::2014-04-11 23:00:50,078::task::827::TaskManager.Task::(resourceAcquired) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive)
>Thread-14::DEBUG::2014-04-11 23:00:50,079::task::990::TaskManager.Task::(_decref) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::ref 1 aborting False
>Thread-14::INFO::2014-04-11 23:00:50,079::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
>Thread-14::DEBUG::2014-04-11 23:00:50,080::task::1180::TaskManager.Task::(prepare) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Prepare: 1 jobs exist, move to acquiring
>Thread-14::DEBUG::2014-04-11 23:00:50,080::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state preparing -> state acquiring
>Thread-14::DEBUG::2014-04-11 23:00:50,080::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state acquiring -> state queued
>Thread-14::DEBUG::2014-04-11 23:00:50,080::taskManager::50::TaskManager::(_queueTask) queuing task: 8be642d5-95c4-4237-9639-a2844d95ee69
>Thread-14::DEBUG::2014-04-11 23:00:50,081::taskManager::56::TaskManager::(_queueTask) task queued: 8be642d5-95c4-4237-9639-a2844d95ee69
>9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:50,081::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
>Thread-14::DEBUG::2014-04-11 23:00:50,083::task::1182::TaskManager.Task::(prepare) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::returning
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,084::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 8be642d5-95c4-4237-9639-a2844d95ee69 running: <bound method Task.commit of <storage.task.Task instance at 0x7fc3f8169830>> with: None
>Thread-14::DEBUG::2014-04-11 23:00:50,085::task::990::TaskManager.Task::(_decref) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::ref 0 aborting False
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,086::task::1194::TaskManager.Task::(commit) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::committing task: 8be642d5-95c4-4237-9639-a2844d95ee69
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,088::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state queued -> state running
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,090::task::905::TaskManager.Task::(_runJobs) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {})
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,092::task::331::TaskManager.Task::(run) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,092::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,093::misc::764::SamplingMethod::(__call__) Got in to sampling method
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,111::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-39::DEBUG::2014-04-11 23:00:50,123::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,126::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,129::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>Thread-39::WARNING::2014-04-11 23:00:50,131::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,134::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-39::DEBUG::2014-04-11 23:00:50,136::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-39::ERROR::2014-04-11 23:00:50,170::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 23:00:50,171::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,187::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,187::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,203::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,204::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,205::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,205::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,221::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,222::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`2977357f-55fd-4820-895a-a161723b8c1e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,223::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,223::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,223::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`2977357f-55fd-4820-895a-a161723b8c1e`::Granted request
>8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,224::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a`
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,225::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`7374d821-417d-42e1-9905-7af8bfa8600a`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,225::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,226::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,226::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`7374d821-417d-42e1-9905-7af8bfa8600a`::Granted request
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,266::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,267::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,284::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,287::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,287::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered
>8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,287::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,288::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,288::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,289::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,289::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,289::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,305::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,306::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,306::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
>Thread-498::DEBUG::2014-04-11 23:00:50,308::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-499::DEBUG::2014-04-11 23:00:50,310::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-500::DEBUG::2014-04-11 23:00:50,314::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-501::ERROR::2014-04-11 23:00:50,319::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-502::ERROR::2014-04-11 23:00:50,324::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-501::ERROR::2014-04-11 23:00:50,328::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-503::DEBUG::2014-04-11 23:00:50,330::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 21619c8e-99ea-4813-be02-d708971e5393
>Thread-498::DEBUG::2014-04-11 23:00:50,331::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`60963b15-5eb8-487c-b63a-903638f2e361`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-502::ERROR::2014-04-11 23:00:50,332::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,334::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-499::DEBUG::2014-04-11 23:00:50,335::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`c8663709-ef61-4309-9fbf-fae6c5053f49`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-500::DEBUG::2014-04-11 23:00:50,339::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`4274a941-8ceb-4b3f-ab73-1a196c9b1a57`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>Thread-498::DEBUG::2014-04-11 23:00:50,342::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive'
>Thread-501::DEBUG::2014-04-11 23:00:50,342::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-499::DEBUG::2014-04-11 23:00:50,348::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,349::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-500::DEBUG::2014-04-11 23:00:50,350::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive'
>Thread-503::DEBUG::2014-04-11 23:00:50,352::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`5bc81f0a-b542-4901-9a5b-8164e05176cb`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,358::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-502::DEBUG::2014-04-11 23:00:50,361::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-503::DEBUG::2014-04-11 23:00:50,362::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,365::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-498::DEBUG::2014-04-11 23:00:50,372::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user)
>Thread-498::DEBUG::2014-04-11 23:00:50,378::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`60963b15-5eb8-487c-b63a-903638f2e361`::Granted request
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,387::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms
>Thread-499::DEBUG::2014-04-11 23:00:50,388::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user)
>Thread-501::DEBUG::2014-04-11 23:00:50,389::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-499::DEBUG::2014-04-11 23:00:50,408::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`c8663709-ef61-4309-9fbf-fae6c5053f49`::Granted request
>Thread-502::DEBUG::2014-04-11 23:00:50,418::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-498::DEBUG::2014-04-11 23:00:50,420::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`2db1388a-4df6-4f17-bf71-271537b05f2e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,420::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks
>Thread-500::DEBUG::2014-04-11 23:00:50,424::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user)
>Thread-498::DEBUG::2014-04-11 23:00:50,441::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive'
>Thread-500::DEBUG::2014-04-11 23:00:50,448::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`4274a941-8ceb-4b3f-ab73-1a196c9b1a57`::Granted request
>Thread-499::DEBUG::2014-04-11 23:00:50,453::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`139ca213-17f3-4c3a-b8ea-63e7f5bf8511`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-503::DEBUG::2014-04-11 23:00:50,458::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user)
>Thread-499::DEBUG::2014-04-11 23:00:50,461::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive'
>Thread-503::DEBUG::2014-04-11 23:00:50,465::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`5bc81f0a-b542-4901-9a5b-8164e05176cb`::Granted request
>Thread-500::DEBUG::2014-04-11 23:00:50,471::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`26a15518-3c6f-4405-987e-44a39be5a581`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-498::DEBUG::2014-04-11 23:00:50,472::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user)
>Thread-500::DEBUG::2014-04-11 23:00:50,474::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive'
>Thread-498::DEBUG::2014-04-11 23:00:50,476::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`2db1388a-4df6-4f17-bf71-271537b05f2e`::Granted request
>Thread-503::DEBUG::2014-04-11 23:00:50,479::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`14c6b8c9-f764-4cbd-bdb2-30e053e94814`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
>Thread-499::DEBUG::2014-04-11 23:00:50,481::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user)
>Thread-503::DEBUG::2014-04-11 23:00:50,485::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive'
>Thread-499::DEBUG::2014-04-11 23:00:50,489::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`139ca213-17f3-4c3a-b8ea-63e7f5bf8511`::Granted request
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,496::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6']
>Thread-500::DEBUG::2014-04-11 23:00:50,498::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user)
>Thread-499::DEBUG::2014-04-11 23:00:50,504::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` is not a data domain it is an 2 domain, not upgrading
>Thread-500::DEBUG::2014-04-11 23:00:50,504::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`26a15518-3c6f-4405-987e-44a39be5a581`::Granted request
>Thread-499::DEBUG::2014-04-11 23:00:50,510::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87'
>Thread-503::DEBUG::2014-04-11 23:00:50,511::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user)
>Thread-503::DEBUG::2014-04-11 23:00:50,517::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`14c6b8c9-f764-4cbd-bdb2-30e053e94814`::Granted request
>Thread-499::DEBUG::2014-04-11 23:00:50,525::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users)
>Thread-498::DEBUG::2014-04-11 23:00:50,526::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955']
>Thread-499::DEBUG::2014-04-11 23:00:50,530::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it.
>Thread-499::DEBUG::2014-04-11 23:00:50,535::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records.
>Thread-498::DEBUG::2014-04-11 23:00:50,542::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-499::DEBUG::2014-04-11 23:00:50,551::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87'
>Thread-498::DEBUG::2014-04-11 23:00:50,552::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-500::DEBUG::2014-04-11 23:00:50,554::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40']
>Thread-498::DEBUG::2014-04-11 23:00:50,560::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-498::DEBUG::2014-04-11 23:00:50,568::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-500::DEBUG::2014-04-11 23:00:50,570::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce'
>Thread-503::DEBUG::2014-04-11 23:00:50,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2']
>Thread-499::DEBUG::2014-04-11 23:00:50,575::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users)
>Thread-499::DEBUG::2014-04-11 23:00:50,582::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it.
>Thread-498::DEBUG::2014-04-11 23:00:50,583::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8'
>Thread-499::DEBUG::2014-04-11 23:00:50,588::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records.
>Thread-503::DEBUG::2014-04-11 23:00:50,588::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393'
>Thread-500::DEBUG::2014-04-11 23:00:50,591::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users)
>Thread-500::DEBUG::2014-04-11 23:00:50,596::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it.
>Thread-500::DEBUG::2014-04-11 23:00:50,596::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records.
>Thread-498::DEBUG::2014-04-11 23:00:50,597::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users)
>Thread-498::DEBUG::2014-04-11 23:00:50,600::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it.
>Thread-500::DEBUG::2014-04-11 23:00:50,600::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce'
>Thread-498::DEBUG::2014-04-11 23:00:50,603::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records.
>Thread-503::DEBUG::2014-04-11 23:00:50,604::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' (0 active users)
>Thread-503::DEBUG::2014-04-11 23:00:50,607::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it.
>Thread-503::DEBUG::2014-04-11 23:00:50,607::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393', Clearing records.
>Thread-500::DEBUG::2014-04-11 23:00:50,608::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users)
>Thread-500::DEBUG::2014-04-11 23:00:50,611::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it.
>Thread-503::DEBUG::2014-04-11 23:00:50,612::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393'
>Thread-500::DEBUG::2014-04-11 23:00:50,613::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records.
>Thread-503::DEBUG::2014-04-11 23:00:50,616::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' (0 active users)
>Thread-503::DEBUG::2014-04-11 23:00:50,619::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it.
>Thread-503::DEBUG::2014-04-11 23:00:50,619::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393', Clearing records.
>Thread-501::DEBUG::2014-04-11 23:00:50,674::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>Thread-501::WARNING::2014-04-11 23:00:50,676::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>Thread-501::ERROR::2014-04-11 23:00:50,709::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:50,711::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:50,713::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,715::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,716::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-502::DEBUG::2014-04-11 23:00:50,734::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-502::WARNING::2014-04-11 23:00:50,745::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-502::ERROR::2014-04-11 23:00:50,779::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,995::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>8be642d5-95c4-4237-9639-a2844d95ee69::WARNING::2014-04-11 23:00:50,997::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,998::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,039::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,040::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
>    self.checkBackupDomain(__securityOverride=True)
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
>    dom = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,042::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,042::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,043::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,044::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:00:51,097::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61]
>Thread-14::DEBUG::2014-04-11 23:00:51,099::task::595::TaskManager.Task::(_updateState) Task=`13eae5eb-8ce5-4e9f-a17d-f264be3a0d1b`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:51,099::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='8be642d5-95c4-4237-9639-a2844d95ee69', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:51,099::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 8be642d5-95c4-4237-9639-a2844d95ee69
>Thread-14::DEBUG::2014-04-11 23:00:51,100::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}
>Thread-14::INFO::2014-04-11 23:00:51,100::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}}
>Thread-14::DEBUG::2014-04-11 23:00:51,101::task::1185::TaskManager.Task::(prepare) Task=`13eae5eb-8ce5-4e9f-a17d-f264be3a0d1b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}}
>Thread-14::DEBUG::2014-04-11 23:00:51,101::task::595::TaskManager.Task::(_updateState) Task=`13eae5eb-8ce5-4e9f-a17d-f264be3a0d1b`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:51,101::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:51,102::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:51,102::task::990::TaskManager.Task::(_decref) Task=`13eae5eb-8ce5-4e9f-a17d-f264be3a0d1b`::ref 0 aborting False
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,311::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5
>8be642d5-95c4-4237-9639-a2844d95ee69::WARNING::2014-04-11 23:00:51,313::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', '  Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb']
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,313::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,346::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,347::sp::329::Storage.StoragePool::(startSpm) Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
>    self._updateDomainsRole()
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
>    domain = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,348::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,348::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,349::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`add08c36-8317-4d8e-b71e-51d8b1644024`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,349::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,350::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,350::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`add08c36-8317-4d8e-b71e-51d8b1644024`::Granted request
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,351::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`44fb3b1f-717b-4b6e-b8c1-36139fb31497`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,351::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' for lock type 'exclusive'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,352::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free. Now locking as 'exclusive' (1 active user)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,352::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`44fb3b1f-717b-4b6e-b8c1-36139fb31497`::Granted request
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,353::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,353::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' (0 active users)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,354::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free, finding out if anyone is waiting for it.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,354::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb', Clearing records.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,355::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`7a624b7c-4ff5-4e26-a1a7-3cea91d6779c`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,355::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' for lock type 'exclusive'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,355::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free. Now locking as 'exclusive' (1 active user)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,356::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`7a624b7c-4ff5-4e26-a1a7-3cea91d6779c`::Granted request
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,356::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,357::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' (0 active users)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,357::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free, finding out if anyone is waiting for it.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,357::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48', Clearing records.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,358::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,358::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,359::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,359::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,360::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,361::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,361::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109']
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,391::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:51,392::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,396::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released
>8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,398::task::866::TaskManager.Task::(_setError) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/storage/task.py", line 334, in run
>    return self.cmd(*self.argslist, **self.argsdict)
>  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
>    self._updateDomainsRole()
>  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
>    return method(self, *args, **kwargs)
>  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
>    domain = sdCache.produce(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,400::task::885::TaskManager.Task::(_run) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Task._run: 8be642d5-95c4-4237-9639-a2844d95ee69 () {} failed - stopping task
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,400::task::1211::TaskManager.Task::(stop) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::stopping in state running (force False)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,401::task::990::TaskManager.Task::(_decref) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::ref 1 aborting True
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,401::task::916::TaskManager.Task::(_runJobs) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,401::task::990::TaskManager.Task::(_decref) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::ref 0 aborting True
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,402::task::925::TaskManager.Task::(_doAbort) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Task._doAbort: force False
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,402::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,402::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state running -> state aborting
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,403::task::550::TaskManager.Task::(__state_aborting) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::_aborting: recover policy auto
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,403::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state aborting -> state racquiring
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,403::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state racquiring -> state recovering
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,404::task::798::TaskManager.Task::(_recover) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::_recover
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,404::task::805::TaskManager.Task::(_recover) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::running recovery None
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,404::task::786::TaskManager.Task::(_recoverDone) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Recover Done: state recovering
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,405::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state recovering -> state recovered
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,405::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>}
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,405::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,406::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,406::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,406::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,407::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,407::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
>Thread-14::DEBUG::2014-04-11 23:00:52,107::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61]
>Thread-14::DEBUG::2014-04-11 23:00:52,108::task::595::TaskManager.Task::(_updateState) Task=`fc0e5f16-24ff-481a-a7db-1b16af7edc6d`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:52,108::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='8be642d5-95c4-4237-9639-a2844d95ee69', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:52,109::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 8be642d5-95c4-4237-9639-a2844d95ee69
>Thread-14::DEBUG::2014-04-11 23:00:52,109::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}
>Thread-14::INFO::2014-04-11 23:00:52,109::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}}
>Thread-14::DEBUG::2014-04-11 23:00:52,109::task::1185::TaskManager.Task::(prepare) Task=`fc0e5f16-24ff-481a-a7db-1b16af7edc6d`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}}
>Thread-14::DEBUG::2014-04-11 23:00:52,110::task::595::TaskManager.Task::(_updateState) Task=`fc0e5f16-24ff-481a-a7db-1b16af7edc6d`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:52,110::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:52,110::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:52,111::task::990::TaskManager.Task::(_decref) Task=`fc0e5f16-24ff-481a-a7db-1b16af7edc6d`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:52,121::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61]
>Thread-14::DEBUG::2014-04-11 23:00:52,121::task::595::TaskManager.Task::(_updateState) Task=`151fcf0b-de1c-4e8c-b8e1-3634038fa8c1`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:52,122::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-14::INFO::2014-04-11 23:00:52,136::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:52,137::task::1185::TaskManager.Task::(prepare) Task=`151fcf0b-de1c-4e8c-b8e1-3634038fa8c1`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-14::DEBUG::2014-04-11 23:00:52,137::task::595::TaskManager.Task::(_updateState) Task=`151fcf0b-de1c-4e8c-b8e1-3634038fa8c1`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:52,137::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:52,137::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:52,138::task::990::TaskManager.Task::(_decref) Task=`151fcf0b-de1c-4e8c-b8e1-3634038fa8c1`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:52,220::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61]
>Thread-14::DEBUG::2014-04-11 23:00:52,220::task::595::TaskManager.Task::(_updateState) Task=`d9812f9c-8b12-4355-ad02-641ff85435ab`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:52,221::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='8be642d5-95c4-4237-9639-a2844d95ee69', spUUID=None, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:52,221::taskManager::161::TaskManager::(clearTask) Entry. taskID: 8be642d5-95c4-4237-9639-a2844d95ee69
>Thread-14::DEBUG::2014-04-11 23:00:52,221::taskManager::166::TaskManager::(clearTask) Return.
>Thread-14::INFO::2014-04-11 23:00:52,222::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
>Thread-14::DEBUG::2014-04-11 23:00:52,222::task::1185::TaskManager.Task::(prepare) Task=`d9812f9c-8b12-4355-ad02-641ff85435ab`::finished: None
>Thread-14::DEBUG::2014-04-11 23:00:52,222::task::595::TaskManager.Task::(_updateState) Task=`d9812f9c-8b12-4355-ad02-641ff85435ab`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:52,222::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:52,223::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:52,223::task::990::TaskManager.Task::(_decref) Task=`d9812f9c-8b12-4355-ad02-641ff85435ab`::ref 0 aborting False
>Thread-43::DEBUG::2014-04-11 23:00:52,764::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-43::DEBUG::2014-04-11 23:00:52,778::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000482084 s, 1.6 MB/s\n'; <rc> = 0
>Thread-24::DEBUG::2014-04-11 23:00:53,591::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-24::DEBUG::2014-04-11 23:00:53,605::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000340935 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2014-04-11 23:00:53,762::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2014-04-11 23:00:53,775::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000528379 s, 1.3 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 23:00:54,084::task::595::TaskManager.Task::(_updateState) Task=`699bf6ac-cbff-4294-b4b1-66b159c7990a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:54,085::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 23:00:54,086::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000528379', 'lastCheck': '0.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340935', 'lastCheck': '0.5', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.0', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000564156', 'lastCheck': '9.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000482084', 'lastCheck': '1.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000302579', 'lastCheck': '9.9', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:54,086::task::1185::TaskManager.Task::(prepare) Task=`699bf6ac-cbff-4294-b4b1-66b159c7990a`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000528379', 'lastCheck': '0.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340935', 'lastCheck': '0.5', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.0', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000564156', 'lastCheck': '9.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000482084', 'lastCheck': '1.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000302579', 'lastCheck': '9.9', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:54,087::task::595::TaskManager.Task::(_updateState) Task=`699bf6ac-cbff-4294-b4b1-66b159c7990a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:54,087::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:54,087::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:54,088::task::990::TaskManager.Task::(_decref) Task=`699bf6ac-cbff-4294-b4b1-66b159c7990a`::ref 0 aborting False
>Thread-29::DEBUG::2014-04-11 23:00:54,175::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-29::DEBUG::2014-04-11 23:00:54,188::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000517063 s, 1.5 MB/s\n'; <rc> = 0
>Thread-46::DEBUG::2014-04-11 23:00:54,227::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 23:00:54,240::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000417934 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14::DEBUG::2014-04-11 23:00:57,224::task::595::TaskManager.Task::(_updateState) Task=`fd1bd315-844d-4f8a-a2a3-fe668e9efd70`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:57,225::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-14::INFO::2014-04-11 23:00:57,225::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000528379', 'lastCheck': '3.4', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340935', 'lastCheck': '3.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000517063', 'lastCheck': '3.0', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000482084', 'lastCheck': '4.4', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000417934', 'lastCheck': '3.0', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:57,226::task::1185::TaskManager.Task::(prepare) Task=`fd1bd315-844d-4f8a-a2a3-fe668e9efd70`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000528379', 'lastCheck': '3.4', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340935', 'lastCheck': '3.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000517063', 'lastCheck': '3.0', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000482084', 'lastCheck': '4.4', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000417934', 'lastCheck': '3.0', 'valid': True}}
>Thread-14::DEBUG::2014-04-11 23:00:57,226::task::595::TaskManager.Task::(_updateState) Task=`fd1bd315-844d-4f8a-a2a3-fe668e9efd70`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:00:57,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:00:57,227::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:00:57,227::task::990::TaskManager.Task::(_decref) Task=`fd1bd315-844d-4f8a-a2a3-fe668e9efd70`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:57,368::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [190fd93d]
>Thread-14::DEBUG::2014-04-11 23:00:57,369::task::595::TaskManager.Task::(_updateState) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:00:57,369::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, remove=False, options=None)
>Thread-14::DEBUG::2014-04-11 23:00:57,370::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d820e0af-8322-4826-a882-f82b53903bf9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1121' at 'disconnectStoragePool'
>Thread-14::DEBUG::2014-04-11 23:00:57,371::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 23:00:57,371::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 23:00:57,371::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d820e0af-8322-4826-a882-f82b53903bf9`::Granted request
>Thread-14::DEBUG::2014-04-11 23:00:57,372::task::827::TaskManager.Task::(resourceAcquired) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive)
>Thread-14::DEBUG::2014-04-11 23:00:57,372::task::990::TaskManager.Task::(_decref) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::ref 1 aborting False
>Thread-14::DEBUG::2014-04-11 23:00:57,373::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`bba216b4-bea8-46ac-8f46-68e7427cc35d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1129' at '_disconnectPool'
>Thread-14::DEBUG::2014-04-11 23:00:57,373::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
>Thread-14::DEBUG::2014-04-11 23:00:57,374::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
>Thread-14::DEBUG::2014-04-11 23:00:57,374::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`bba216b4-bea8-46ac-8f46-68e7427cc35d`::Granted request
>Thread-14::INFO::2014-04-11 23:00:57,375::sp::644::Storage.StoragePool::(disconnect) Disconnect from the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c
>Thread-14::INFO::2014-04-11 23:00:57,375::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-22::DEBUG::2014-04-11 23:00:57,376::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 423b56b3-3148-468c-8c28-4c96fddbc1c8
>Thread-22::INFO::2014-04-11 23:00:57,376::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1)
>Thread-22::DEBUG::2014-04-11 23:00:59,377::clusterlock::217::SANLock::(releaseHostId) Host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 released successfully (id: 1)
>Thread-14::INFO::2014-04-11 23:00:59,378::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-24::DEBUG::2014-04-11 23:00:59,379::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for e2ffd939-380e-4fbb-8a23-5a0e69c84a87
>Thread-14::INFO::2014-04-11 23:00:59,379::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-33::DEBUG::2014-04-11 23:00:59,380::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 3406665e-4adc-4fd4-aa1e-037547b29adb
>Thread-14::INFO::2014-04-11 23:00:59,380::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-29::DEBUG::2014-04-11 23:00:59,381::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for b7663d70-e658-41fa-b9f0-8da83c9eddce
>Thread-29::INFO::2014-04-11 23:00:59,381::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1)
>Thread-39::ERROR::2014-04-11 23:01:00,172::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::ERROR::2014-04-11 23:01:00,173::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 23:01:00,174::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-39::DEBUG::2014-04-11 23:01:00,175::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None)
>Thread-39::DEBUG::2014-04-11 23:01:00,448::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5
>Thread-39::WARNING::2014-04-11 23:01:00,450::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', '  Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48']
>Thread-39::DEBUG::2014-04-11 23:01:00,450::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-39::ERROR::2014-04-11 23:01:00,483::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-39::ERROR::2014-04-11 23:01:00,484::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
>    self.domain = sdCache.produce(self.sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>    domain.getRealDomain()
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>    return self._cache._realProduce(self._sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>    domain = self._findDomain(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>    dom = findMethod(sdUUID)
>  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>    raise se.StorageDomainDoesNotExist(sdUUID)
>StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',)
>Thread-29::DEBUG::2014-04-11 23:01:01,382::clusterlock::217::SANLock::(releaseHostId) Host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce released successfully (id: 1)
>Thread-14::INFO::2014-04-11 23:01:01,383::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-39::DEBUG::2014-04-11 23:01:01,383::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for f3b51811-4a7f-43af-8633-322b3db23c48
>Thread-14::INFO::2014-04-11 23:01:01,384::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-43::DEBUG::2014-04-11 23:01:01,384::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for a033c2ac-0d01-490c-9552-99ca53d6a64a
>Thread-43::INFO::2014-04-11 23:01:01,385::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1)
>Thread-15::DEBUG::2014-04-11 23:01:02,336::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-15::DEBUG::2014-04-11 23:01:02,337::task::595::TaskManager.Task::(_updateState) Task=`0373faeb-5195-4b7e-99d7-5966521c74c9`::moving from state init -> state preparing
>Thread-15::INFO::2014-04-11 23:01:02,338::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None)
>Thread-15::INFO::2014-04-11 23:01:02,355::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-15::DEBUG::2014-04-11 23:01:02,355::task::1185::TaskManager.Task::(prepare) Task=`0373faeb-5195-4b7e-99d7-5966521c74c9`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
>Thread-15::DEBUG::2014-04-11 23:01:02,356::task::595::TaskManager.Task::(_updateState) Task=`0373faeb-5195-4b7e-99d7-5966521c74c9`::moving from state preparing -> state finished
>Thread-15::DEBUG::2014-04-11 23:01:02,356::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15::DEBUG::2014-04-11 23:01:02,356::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15::DEBUG::2014-04-11 23:01:02,357::task::990::TaskManager.Task::(_decref) Task=`0373faeb-5195-4b7e-99d7-5966521c74c9`::ref 0 aborting False
>Thread-15::DEBUG::2014-04-11 23:01:02,361::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1]
>Thread-15::DEBUG::2014-04-11 23:01:02,362::task::595::TaskManager.Task::(_updateState) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::moving from state init -> state preparing
>Thread-15::INFO::2014-04-11 23:01:02,362::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
>Thread-15::ERROR::2014-04-11 23:01:02,363::task::866::TaskManager.Task::(_setError) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::Unexpected error
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/task.py", line 873, in _run
>    return fn(*args, **kargs)
>  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>    res = f(*args, **kwargs)
>  File "/usr/share/vdsm/storage/hsm.py", line 2115, in getAllTasksStatuses
>    allTasksStatus = sp.getAllTasksStatuses()
>  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
>    raise SecureError("Secured object is not in safe state")
>SecureError: Secured object is not in safe state
>Thread-15::DEBUG::2014-04-11 23:01:02,363::task::885::TaskManager.Task::(_run) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::Task._run: 610b39ba-e53e-464a-8cc8-72bf57fbf266 () {} failed - stopping task
>Thread-15::DEBUG::2014-04-11 23:01:02,364::task::1211::TaskManager.Task::(stop) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::stopping in state preparing (force False)
>Thread-15::DEBUG::2014-04-11 23:01:02,364::task::990::TaskManager.Task::(_decref) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::ref 1 aborting True
>Thread-15::INFO::2014-04-11 23:01:02,364::task::1168::TaskManager.Task::(prepare) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
>Thread-15::DEBUG::2014-04-11 23:01:02,365::task::1173::TaskManager.Task::(prepare) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::Prepare: aborted: Secured object is not in safe state
>Thread-15::DEBUG::2014-04-11 23:01:02,365::task::990::TaskManager.Task::(_decref) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::ref 0 aborting True
>Thread-15::DEBUG::2014-04-11 23:01:02,365::task::925::TaskManager.Task::(_doAbort) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::Task._doAbort: force False
>Thread-15::DEBUG::2014-04-11 23:01:02,366::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15::DEBUG::2014-04-11 23:01:02,366::task::595::TaskManager.Task::(_updateState) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::moving from state preparing -> state aborting
>Thread-15::DEBUG::2014-04-11 23:01:02,366::task::550::TaskManager.Task::(__state_aborting) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::_aborting: recover policy none
>Thread-15::DEBUG::2014-04-11 23:01:02,366::task::595::TaskManager.Task::(_updateState) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::moving from state aborting -> state failed
>Thread-15::DEBUG::2014-04-11 23:01:02,367::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15::DEBUG::2014-04-11 23:01:02,367::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15::ERROR::2014-04-11 23:01:02,367::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
>    result = ctask.prepare(self.func, *args, **kwargs)
>  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
>    return m(self, *a, **kw)
>  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
>    raise self.error
>SecureError: Secured object is not in safe state
>Thread-46::DEBUG::2014-04-11 23:01:04,269::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None)
>Thread-46::DEBUG::2014-04-11 23:01:04,282::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000376024 s, 2.0 MB/s\n'; <rc> = 0
>Thread-43::DEBUG::2014-04-11 23:01:04,386::clusterlock::217::SANLock::(releaseHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a released successfully (id: 1)
>Thread-14::INFO::2014-04-11 23:01:04,386::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 21619c8e-99ea-4813-be02-d708971e5393
>Thread-46::DEBUG::2014-04-11 23:01:04,387::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 21619c8e-99ea-4813-be02-d708971e5393
>Thread-46::INFO::2014-04-11 23:01:04,387::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1)
>Thread-46::DEBUG::2014-04-11 23:01:06,388::clusterlock::217::SANLock::(releaseHostId) Host id for domain 21619c8e-99ea-4813-be02-d708971e5393 released successfully (id: 1)
>Thread-14::DEBUG::2014-04-11 23:01:06,390::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
>Thread-14::DEBUG::2014-04-11 23:01:06,390::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
>Thread-14::DEBUG::2014-04-11 23:01:06,391::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 23:01:06,391::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
>Thread-14::INFO::2014-04-11 23:01:06,392::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: True
>Thread-14::DEBUG::2014-04-11 23:01:06,392::task::1185::TaskManager.Task::(prepare) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::finished: True
>Thread-14::DEBUG::2014-04-11 23:01:06,392::task::595::TaskManager.Task::(_updateState) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:01:06,393::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>}
>Thread-14::DEBUG::2014-04-11 23:01:06,393::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:01:06,394::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c'
>Thread-14::DEBUG::2014-04-11 23:01:06,394::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users)
>Thread-14::DEBUG::2014-04-11 23:01:06,394::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it.
>Thread-14::DEBUG::2014-04-11 23:01:06,395::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records.
>Thread-14::DEBUG::2014-04-11 23:01:06,395::task::990::TaskManager.Task::(_decref) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:01:06,473::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [159e2882]
>Thread-14::DEBUG::2014-04-11 23:01:06,473::task::595::TaskManager.Task::(_updateState) Task=`c4b2385a-71b2-410b-b35d-11508237367c`::moving from state init -> state preparing
>Thread-14::INFO::2014-04-11 23:01:06,474::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', conList=[{'connection': 'ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c', 'port': ''}, {'connection': 'ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/exports', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54', 'port': ''}, {'connection': 'ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'deb06d0b-14c6-490e-8fbb-552861511109', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/infrastructure', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac', 'port': ''}], options=None)
>Thread-14::DEBUG::2014-04-11 23:01:06,475::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:01:06,583::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:01:06,640::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_exports' (cwd None)
>Thread-14::ERROR::2014-04-11 23:01:06,664::hsm::2477::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2473, in disconnectStorageServer
>    conObj.disconnect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect
>    return self._mountCon.disconnect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect
>    self._mount.umount(True, True)
>  File "/usr/share/vdsm/storage/mount.py", line 253, in umount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (1, ';umount: /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_exports: not found\n')
>Thread-14::DEBUG::2014-04-11 23:01:06,666::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:01:06,709::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure' (cwd None)
>Thread-14::ERROR::2014-04-11 23:01:06,746::hsm::2477::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/hsm.py", line 2473, in disconnectStorageServer
>    conObj.disconnect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect
>    return self._mountCon.disconnect()
>  File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect
>    self._mount.umount(True, True)
>  File "/usr/share/vdsm/storage/mount.py", line 253, in umount
>    return self._runcmd(cmd, timeout)
>  File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd
>    raise MountError(rc, ";".join((out, err)))
>MountError: (1, ';umount: /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure: not found\n')
>Thread-14::DEBUG::2014-04-11 23:01:06,747::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:01:06,791::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:01:06,830::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-14::DEBUG::2014-04-11 23:01:06,830::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 23:01:06,831::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-14::DEBUG::2014-04-11 23:01:06,831::misc::764::SamplingMethod::(__call__) Got in to sampling method
>Thread-14::DEBUG::2014-04-11 23:01:06,832::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
>Thread-14::DEBUG::2014-04-11 23:01:06,832::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:01:06,856::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::DEBUG::2014-04-11 23:01:06,856::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
>Thread-14::DEBUG::2014-04-11 23:01:07,032::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
>Thread-14::DEBUG::2014-04-11 23:01:07,033::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 23:01:07,034::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 23:01:07,035::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 23:01:07,035::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 23:01:07,036::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-14::DEBUG::2014-04-11 23:01:07,036::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-14::DEBUG::2014-04-11 23:01:07,036::misc::772::SamplingMethod::(__call__) Returning last result
>Thread-14::INFO::2014-04-11 23:01:07,037::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]}
>Thread-14::DEBUG::2014-04-11 23:01:07,038::task::1185::TaskManager.Task::(prepare) Task=`c4b2385a-71b2-410b-b35d-11508237367c`::finished: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]}
>Thread-14::DEBUG::2014-04-11 23:01:07,038::task::595::TaskManager.Task::(_updateState) Task=`c4b2385a-71b2-410b-b35d-11508237367c`::moving from state preparing -> state finished
>Thread-14::DEBUG::2014-04-11 23:01:07,039::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-14::DEBUG::2014-04-11 23:01:07,039::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-14::DEBUG::2014-04-11 23:01:07,039::task::990::TaskManager.Task::(_decref) Task=`c4b2385a-71b2-410b-b35d-11508237367c`::ref 0 aborting False
>Thread-14::DEBUG::2014-04-11 23:02:27,065::API::1134::vds::(fenceNode) fenceNode(addr=172.16.10.6,port=,agent=ipmilan,user=root,passwd=XXXX,action=status,secure=,options=lanplus)
>Thread-14::DEBUG::2014-04-11 23:02:27,140::API::1160::vds::(fenceNode) rc 0 in agent=fence_ipmilan
>ipaddr=172.16.10.6
>login=root
>action=status
>passwd=XXXX
>lanplus out Getting status of IPMI:172.16.10.6...Chassis power = On
>Done
> err 
>Thread-14::DEBUG::2014-04-11 23:02:27,148::API::1134::vds::(fenceNode) fenceNode(addr=172.16.10.6,port=,agent=ipmilan,user=root,passwd=XXXX,action=status,secure=,options=lanplus)
>Thread-14::DEBUG::2014-04-11 23:02:27,252::API::1160::vds::(fenceNode) rc 0 in agent=fence_ipmilan
>ipaddr=172.16.10.6
>login=root
>action=status
>passwd=XXXX
>lanplus out Getting status of IPMI:172.16.10.6...Chassis power = On
>Done
> err

Actions: View

Attachments on bug 1086951: 885644 | 885645 | 885646