Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 885644 Details for
Bug 1086951
SPM never stops contending
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
VDSM Log
vdsm.log (text/plain), 2.11 MB, created by
Maurice James
on 2014-04-12 03:12:08 UTC
(
hide
)
Description:
VDSM Log
Filename:
MIME Type:
Creator:
Maurice James
Created:
2014-04-12 03:12:08 UTC
Size:
2.11 MB
patch
obsolete
>Thread-22::DEBUG::2014-04-11 22:01:01,888::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:01:01,910::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000340689 s, 1.1 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:02,105::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:02,106::task::595::TaskManager.Task::(_updateState) Task=`d5fe3e2f-b47c-4bce-a990-d0e03035d5af`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:02,106::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:02,106::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:02,107::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:02,107::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:02,107::task::1185::TaskManager.Task::(prepare) Task=`d5fe3e2f-b47c-4bce-a990-d0e03035d5af`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:02,108::task::595::TaskManager.Task::(_updateState) Task=`d5fe3e2f-b47c-4bce-a990-d0e03035d5af`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:02,108::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:02,108::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:02,109::task::990::TaskManager.Task::(_decref) Task=`d5fe3e2f-b47c-4bce-a990-d0e03035d5af`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:03,113::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:03,141::task::595::TaskManager.Task::(_updateState) Task=`60f26a97-6c95-45cc-888e-d3f1ec3b9a8b`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:03,142::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:03,142::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:03,142::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:03,143::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:03,143::task::1185::TaskManager.Task::(prepare) Task=`60f26a97-6c95-45cc-888e-d3f1ec3b9a8b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:03,143::task::595::TaskManager.Task::(_updateState) Task=`60f26a97-6c95-45cc-888e-d3f1ec3b9a8b`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:03,144::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:03,144::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:03,144::task::990::TaskManager.Task::(_decref) Task=`60f26a97-6c95-45cc-888e-d3f1ec3b9a8b`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:01:03,395::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:01:03,407::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000293048 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:04,149::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:04,150::task::595::TaskManager.Task::(_updateState) Task=`3c4d349a-5389-41d7-a189-fc599da780ef`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:04,150::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:04,151::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:04,151::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:04,151::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:04,152::task::1185::TaskManager.Task::(prepare) Task=`3c4d349a-5389-41d7-a189-fc599da780ef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:04,152::task::595::TaskManager.Task::(_updateState) Task=`3c4d349a-5389-41d7-a189-fc599da780ef`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:04,153::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:04,153::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:04,154::task::990::TaskManager.Task::(_decref) Task=`3c4d349a-5389-41d7-a189-fc599da780ef`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:04,961::task::595::TaskManager.Task::(_updateState) Task=`3c94ac1d-e613-461d-8597-3213799f8883`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:04,961::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-13::INFO::2014-04-11 22:01:04,962::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '33.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340689', 'lastCheck': '3.0', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.3', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000565291', 'lastCheck': '5.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.3', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '12.0', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000293048', 'lastCheck': '1.5', 'valid': True}} >Thread-13::DEBUG::2014-04-11 22:01:04,963::task::1185::TaskManager.Task::(prepare) Task=`3c94ac1d-e613-461d-8597-3213799f8883`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '33.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340689', 'lastCheck': '3.0', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.3', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000565291', 'lastCheck': '5.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.3', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '12.0', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000293048', 'lastCheck': '1.5', 'valid': True}} >Thread-13::DEBUG::2014-04-11 22:01:04,963::task::595::TaskManager.Task::(_updateState) Task=`3c94ac1d-e613-461d-8597-3213799f8883`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:04,963::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:04,964::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:04,964::task::990::TaskManager.Task::(_decref) Task=`3c94ac1d-e613-461d-8597-3213799f8883`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:05,158::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:05,158::task::595::TaskManager.Task::(_updateState) Task=`b04d6d8c-db82-48ef-9fda-034326fe277e`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:05,159::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:05,159::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:05,160::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:05,160::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:05,161::task::1185::TaskManager.Task::(prepare) Task=`b04d6d8c-db82-48ef-9fda-034326fe277e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:05,161::task::595::TaskManager.Task::(_updateState) Task=`b04d6d8c-db82-48ef-9fda-034326fe277e`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:05,162::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:05,162::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:05,162::task::990::TaskManager.Task::(_decref) Task=`b04d6d8c-db82-48ef-9fda-034326fe277e`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:06,167::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:06,168::task::595::TaskManager.Task::(_updateState) Task=`fd6437d2-54c1-4603-b4e1-85c37e05b865`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:06,169::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:06,169::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:06,170::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:06,170::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:06,170::task::1185::TaskManager.Task::(prepare) Task=`fd6437d2-54c1-4603-b4e1-85c37e05b865`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:06,171::task::595::TaskManager.Task::(_updateState) Task=`fd6437d2-54c1-4603-b4e1-85c37e05b865`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:06,171::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:06,171::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:06,171::task::990::TaskManager.Task::(_decref) Task=`fd6437d2-54c1-4603-b4e1-85c37e05b865`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:07,176::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:07,177::task::595::TaskManager.Task::(_updateState) Task=`c3f2b873-94bc-40e4-ae78-a990b29f8925`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:07,177::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:07,178::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:07,178::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:07,178::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:07,178::task::1185::TaskManager.Task::(prepare) Task=`c3f2b873-94bc-40e4-ae78-a990b29f8925`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:07,179::task::595::TaskManager.Task::(_updateState) Task=`c3f2b873-94bc-40e4-ae78-a990b29f8925`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:07,179::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:07,179::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:07,180::task::990::TaskManager.Task::(_decref) Task=`c3f2b873-94bc-40e4-ae78-a990b29f8925`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:08,183::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:08,184::task::595::TaskManager.Task::(_updateState) Task=`a99a7389-c991-4613-9c5b-e3fda4746ab0`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:08,185::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:08,185::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:08,185::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:08,186::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:08,186::task::1185::TaskManager.Task::(prepare) Task=`a99a7389-c991-4613-9c5b-e3fda4746ab0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:08,186::task::595::TaskManager.Task::(_updateState) Task=`a99a7389-c991-4613-9c5b-e3fda4746ab0`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:08,186::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:08,187::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:08,187::task::990::TaskManager.Task::(_decref) Task=`a99a7389-c991-4613-9c5b-e3fda4746ab0`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:09,191::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:09,192::task::595::TaskManager.Task::(_updateState) Task=`e33f762b-dc4a-4768-928e-141b364c5a9d`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:09,192::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:09,192::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:09,193::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:09,193::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:09,193::task::1185::TaskManager.Task::(prepare) Task=`e33f762b-dc4a-4768-928e-141b364c5a9d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:09,194::task::595::TaskManager.Task::(_updateState) Task=`e33f762b-dc4a-4768-928e-141b364c5a9d`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:09,194::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:09,194::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:09,194::task::990::TaskManager.Task::(_decref) Task=`e33f762b-dc4a-4768-928e-141b364c5a9d`::ref 0 aborting False >Thread-25::DEBUG::2014-04-11 22:01:09,808::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-25::DEBUG::2014-04-11 22:01:09,823::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000546938 s, 1.4 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:10,200::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:10,201::task::595::TaskManager.Task::(_updateState) Task=`faec843c-6b99-4781-9636-93697727aed9`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:10,201::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:10,201::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:10,202::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:10,202::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:10,203::task::1185::TaskManager.Task::(prepare) Task=`faec843c-6b99-4781-9636-93697727aed9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:10,203::task::595::TaskManager.Task::(_updateState) Task=`faec843c-6b99-4781-9636-93697727aed9`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:10,204::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:10,204::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:10,204::task::990::TaskManager.Task::(_decref) Task=`faec843c-6b99-4781-9636-93697727aed9`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:11,209::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:11,210::task::595::TaskManager.Task::(_updateState) Task=`5cd13cef-032b-4357-843f-b5e75ed6d160`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:11,210::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:11,211::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:11,211::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:11,211::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:11,212::task::1185::TaskManager.Task::(prepare) Task=`5cd13cef-032b-4357-843f-b5e75ed6d160`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:11,212::task::595::TaskManager.Task::(_updateState) Task=`5cd13cef-032b-4357-843f-b5e75ed6d160`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:11,212::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:11,213::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:11,214::task::990::TaskManager.Task::(_decref) Task=`5cd13cef-032b-4357-843f-b5e75ed6d160`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:01:11,946::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:01:11,961::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000337815 s, 1.1 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:12,257::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:12,258::task::595::TaskManager.Task::(_updateState) Task=`1a1642e5-2a6c-465a-8c58-65c1178aa44f`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:12,259::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:12,259::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:12,260::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:12,260::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:12,260::task::1185::TaskManager.Task::(prepare) Task=`1a1642e5-2a6c-465a-8c58-65c1178aa44f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:12,261::task::595::TaskManager.Task::(_updateState) Task=`1a1642e5-2a6c-465a-8c58-65c1178aa44f`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:12,261::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:12,261::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:12,261::task::990::TaskManager.Task::(_decref) Task=`1a1642e5-2a6c-465a-8c58-65c1178aa44f`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:13,266::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:13,269::task::595::TaskManager.Task::(_updateState) Task=`18062adc-e410-449a-a197-8f65ec3661d2`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:13,269::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:13,270::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:13,270::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:13,270::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:13,271::task::1185::TaskManager.Task::(prepare) Task=`18062adc-e410-449a-a197-8f65ec3661d2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:13,271::task::595::TaskManager.Task::(_updateState) Task=`18062adc-e410-449a-a197-8f65ec3661d2`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:13,271::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:13,271::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:13,272::task::990::TaskManager.Task::(_decref) Task=`18062adc-e410-449a-a197-8f65ec3661d2`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:01:13,434::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:01:13,449::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000239907 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:14,276::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:14,277::task::595::TaskManager.Task::(_updateState) Task=`1e9bea24-d58e-4f1d-859b-0b0919f315d6`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:14,278::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:14,278::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:14,279::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:14,279::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:14,280::task::1185::TaskManager.Task::(prepare) Task=`1e9bea24-d58e-4f1d-859b-0b0919f315d6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:14,280::task::595::TaskManager.Task::(_updateState) Task=`1e9bea24-d58e-4f1d-859b-0b0919f315d6`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:14,281::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:14,281::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:14,281::task::990::TaskManager.Task::(_decref) Task=`1e9bea24-d58e-4f1d-859b-0b0919f315d6`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:15,285::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:15,286::task::595::TaskManager.Task::(_updateState) Task=`cfbb1c02-5c4c-42a1-810b-67e802ecc3c9`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:15,286::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:15,287::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:15,287::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:15,287::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:15,288::task::1185::TaskManager.Task::(prepare) Task=`cfbb1c02-5c4c-42a1-810b-67e802ecc3c9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:15,288::task::595::TaskManager.Task::(_updateState) Task=`cfbb1c02-5c4c-42a1-810b-67e802ecc3c9`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:15,288::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:15,289::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:15,289::task::990::TaskManager.Task::(_decref) Task=`cfbb1c02-5c4c-42a1-810b-67e802ecc3c9`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:16,294::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:16,295::task::595::TaskManager.Task::(_updateState) Task=`da9cbcdb-c143-427b-84e0-cc8012faa320`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:16,296::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:16,296::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:16,297::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:16,297::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:16,297::task::1185::TaskManager.Task::(prepare) Task=`da9cbcdb-c143-427b-84e0-cc8012faa320`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:16,298::task::595::TaskManager.Task::(_updateState) Task=`da9cbcdb-c143-427b-84e0-cc8012faa320`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:16,298::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:16,299::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:16,299::task::990::TaskManager.Task::(_decref) Task=`da9cbcdb-c143-427b-84e0-cc8012faa320`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:17,304::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:17,304::task::595::TaskManager.Task::(_updateState) Task=`a7f71b8b-f14f-40d2-9d29-7bf308cda21c`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:17,305::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:17,305::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:17,306::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:17,306::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:17,307::task::1185::TaskManager.Task::(prepare) Task=`a7f71b8b-f14f-40d2-9d29-7bf308cda21c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:17,307::task::595::TaskManager.Task::(_updateState) Task=`a7f71b8b-f14f-40d2-9d29-7bf308cda21c`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:17,307::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:17,308::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:17,308::task::990::TaskManager.Task::(_decref) Task=`a7f71b8b-f14f-40d2-9d29-7bf308cda21c`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:18,352::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:18,354::task::595::TaskManager.Task::(_updateState) Task=`d38824b0-132b-447b-8201-8c6482c58225`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:18,354::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:18,354::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:18,355::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:18,355::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:18,355::task::1185::TaskManager.Task::(prepare) Task=`d38824b0-132b-447b-8201-8c6482c58225`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:18,356::task::595::TaskManager.Task::(_updateState) Task=`d38824b0-132b-447b-8201-8c6482c58225`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:18,356::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:18,356::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:18,356::task::990::TaskManager.Task::(_decref) Task=`d38824b0-132b-447b-8201-8c6482c58225`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:19,360::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:19,361::task::595::TaskManager.Task::(_updateState) Task=`25510db7-a778-4d84-8bbd-c9b2da6157f8`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:19,361::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:19,361::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:19,362::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:19,362::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:19,362::task::1185::TaskManager.Task::(prepare) Task=`25510db7-a778-4d84-8bbd-c9b2da6157f8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:19,363::task::595::TaskManager.Task::(_updateState) Task=`25510db7-a778-4d84-8bbd-c9b2da6157f8`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:19,363::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:19,363::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:19,364::task::990::TaskManager.Task::(_decref) Task=`25510db7-a778-4d84-8bbd-c9b2da6157f8`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:20,319::task::595::TaskManager.Task::(_updateState) Task=`9407ce89-7b11-4fa4-bd20-62644f2b46f0`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:20,319::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-13::INFO::2014-04-11 22:01:20,320::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '49.1', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000337815', 'lastCheck': '8.4', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '10.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.7', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '27.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000239907', 'lastCheck': '6.9', 'valid': True}} >Thread-13::DEBUG::2014-04-11 22:01:20,320::task::1185::TaskManager.Task::(prepare) Task=`9407ce89-7b11-4fa4-bd20-62644f2b46f0`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '49.1', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000337815', 'lastCheck': '8.4', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '10.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.7', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '27.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000239907', 'lastCheck': '6.9', 'valid': True}} >Thread-13::DEBUG::2014-04-11 22:01:20,321::task::595::TaskManager.Task::(_updateState) Task=`9407ce89-7b11-4fa4-bd20-62644f2b46f0`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:20,321::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:20,321::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:20,322::task::990::TaskManager.Task::(_decref) Task=`9407ce89-7b11-4fa4-bd20-62644f2b46f0`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:20,371::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:20,372::task::595::TaskManager.Task::(_updateState) Task=`fdd6de9d-916f-4d5b-9d6a-cc5bcc05dc4c`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:20,372::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:20,372::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:20,373::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:20,373::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:20,373::task::1185::TaskManager.Task::(prepare) Task=`fdd6de9d-916f-4d5b-9d6a-cc5bcc05dc4c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:20,373::task::595::TaskManager.Task::(_updateState) Task=`fdd6de9d-916f-4d5b-9d6a-cc5bcc05dc4c`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:20,374::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:20,374::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:20,374::task::990::TaskManager.Task::(_decref) Task=`fdd6de9d-916f-4d5b-9d6a-cc5bcc05dc4c`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:21,418::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:21,419::task::595::TaskManager.Task::(_updateState) Task=`1098356d-057a-40e2-9ab7-a2403466a51e`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:21,419::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:21,420::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:21,420::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:21,420::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:21,421::task::1185::TaskManager.Task::(prepare) Task=`1098356d-057a-40e2-9ab7-a2403466a51e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:21,421::task::595::TaskManager.Task::(_updateState) Task=`1098356d-057a-40e2-9ab7-a2403466a51e`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:21,421::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:21,422::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:21,422::task::990::TaskManager.Task::(_decref) Task=`1098356d-057a-40e2-9ab7-a2403466a51e`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:01:21,985::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:01:21,999::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000277253 s, 1.3 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:22,427::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:22,428::task::595::TaskManager.Task::(_updateState) Task=`0ff766f9-56cc-49c1-85d7-8bc9572ada49`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:22,429::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:22,429::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:22,430::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:22,430::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:22,430::task::1185::TaskManager.Task::(prepare) Task=`0ff766f9-56cc-49c1-85d7-8bc9572ada49`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:22,431::task::595::TaskManager.Task::(_updateState) Task=`0ff766f9-56cc-49c1-85d7-8bc9572ada49`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:22,431::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:22,432::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:22,432::task::990::TaskManager.Task::(_decref) Task=`0ff766f9-56cc-49c1-85d7-8bc9572ada49`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:23,436::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:23,453::task::595::TaskManager.Task::(_updateState) Task=`c6548bf0-5af2-4cf7-a657-6c6ce3ddae34`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:23,454::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:23,456::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:23,460::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:23,464::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:23,468::task::1185::TaskManager.Task::(prepare) Task=`c6548bf0-5af2-4cf7-a657-6c6ce3ddae34`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:23,472::task::595::TaskManager.Task::(_updateState) Task=`c6548bf0-5af2-4cf7-a657-6c6ce3ddae34`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:23,476::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:23,479::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:23,481::task::990::TaskManager.Task::(_decref) Task=`c6548bf0-5af2-4cf7-a657-6c6ce3ddae34`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:01:23,516::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:01:23,541::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000250886 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:24,494::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:24,495::task::595::TaskManager.Task::(_updateState) Task=`8e8f7582-00ac-4a11-914a-6740ed11b0d8`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:24,495::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:24,495::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:24,496::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:24,496::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:24,496::task::1185::TaskManager.Task::(prepare) Task=`8e8f7582-00ac-4a11-914a-6740ed11b0d8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:24,497::task::595::TaskManager.Task::(_updateState) Task=`8e8f7582-00ac-4a11-914a-6740ed11b0d8`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:24,497::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:24,497::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:24,498::task::990::TaskManager.Task::(_decref) Task=`8e8f7582-00ac-4a11-914a-6740ed11b0d8`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:25,502::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:25,502::task::595::TaskManager.Task::(_updateState) Task=`e7bdb12a-4f46-4b80-905e-7d0f43ff74b0`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:25,503::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:25,503::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:25,503::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:25,504::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:25,504::task::1185::TaskManager.Task::(prepare) Task=`e7bdb12a-4f46-4b80-905e-7d0f43ff74b0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:25,504::task::595::TaskManager.Task::(_updateState) Task=`e7bdb12a-4f46-4b80-905e-7d0f43ff74b0`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:25,504::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:25,505::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:25,505::task::990::TaskManager.Task::(_decref) Task=`e7bdb12a-4f46-4b80-905e-7d0f43ff74b0`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:26,511::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:26,513::task::595::TaskManager.Task::(_updateState) Task=`b25f7ec8-75bb-4bdf-ba06-37d9381d55e9`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:26,513::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:26,513::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:26,514::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:26,514::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:26,515::task::1185::TaskManager.Task::(prepare) Task=`b25f7ec8-75bb-4bdf-ba06-37d9381d55e9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:26,515::task::595::TaskManager.Task::(_updateState) Task=`b25f7ec8-75bb-4bdf-ba06-37d9381d55e9`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:26,515::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:26,516::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:26,516::task::990::TaskManager.Task::(_decref) Task=`b25f7ec8-75bb-4bdf-ba06-37d9381d55e9`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:27,521::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:27,522::task::595::TaskManager.Task::(_updateState) Task=`17592de4-b554-462c-8972-d7b7090934e4`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:27,522::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:27,522::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:27,523::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:27,523::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:27,523::task::1185::TaskManager.Task::(prepare) Task=`17592de4-b554-462c-8972-d7b7090934e4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:27,524::task::595::TaskManager.Task::(_updateState) Task=`17592de4-b554-462c-8972-d7b7090934e4`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:27,524::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:27,525::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:27,525::task::990::TaskManager.Task::(_decref) Task=`17592de4-b554-462c-8972-d7b7090934e4`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:28,529::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:28,530::task::595::TaskManager.Task::(_updateState) Task=`4c02caf8-14ee-4659-a8b2-8ede5bf42487`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:28,530::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:28,530::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:28,531::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:28,531::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:28,531::task::1185::TaskManager.Task::(prepare) Task=`4c02caf8-14ee-4659-a8b2-8ede5bf42487`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:28,531::task::595::TaskManager.Task::(_updateState) Task=`4c02caf8-14ee-4659-a8b2-8ede5bf42487`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:28,532::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:28,532::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:28,532::task::990::TaskManager.Task::(_decref) Task=`4c02caf8-14ee-4659-a8b2-8ede5bf42487`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:29,537::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:29,538::task::595::TaskManager.Task::(_updateState) Task=`6345a132-f550-4d4b-8c42-bc2c44ccb23c`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:29,538::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:29,538::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:29,539::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:29,539::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:29,540::task::1185::TaskManager.Task::(prepare) Task=`6345a132-f550-4d4b-8c42-bc2c44ccb23c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:29,540::task::595::TaskManager.Task::(_updateState) Task=`6345a132-f550-4d4b-8c42-bc2c44ccb23c`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:29,541::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:29,541::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:29,541::task::990::TaskManager.Task::(_decref) Task=`6345a132-f550-4d4b-8c42-bc2c44ccb23c`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:30,545::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:30,546::task::595::TaskManager.Task::(_updateState) Task=`da13f63b-45a6-4db1-9e68-68f2a0d96cb4`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:30,546::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:30,547::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:30,547::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:30,547::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:30,548::task::1185::TaskManager.Task::(prepare) Task=`da13f63b-45a6-4db1-9e68-68f2a0d96cb4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:30,548::task::595::TaskManager.Task::(_updateState) Task=`da13f63b-45a6-4db1-9e68-68f2a0d96cb4`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:30,549::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:30,549::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:30,549::task::990::TaskManager.Task::(_decref) Task=`da13f63b-45a6-4db1-9e68-68f2a0d96cb4`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:31,555::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:31,556::task::595::TaskManager.Task::(_updateState) Task=`8afc502f-a6c0-4cb2-b90b-070e80434c32`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:31,557::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:31,557::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:31,558::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:31,559::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:31,559::task::1185::TaskManager.Task::(prepare) Task=`8afc502f-a6c0-4cb2-b90b-070e80434c32`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:31,560::task::595::TaskManager.Task::(_updateState) Task=`8afc502f-a6c0-4cb2-b90b-070e80434c32`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:31,561::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:31,561::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:31,561::task::990::TaskManager.Task::(_decref) Task=`8afc502f-a6c0-4cb2-b90b-070e80434c32`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:01:32,038::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:01:32,054::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000273063 s, 1.4 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:32,567::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:32,570::task::595::TaskManager.Task::(_updateState) Task=`23ee456d-fd6b-48c2-9e03-932941e20344`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:32,571::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:32,571::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:32,572::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:32,572::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:32,572::task::1185::TaskManager.Task::(prepare) Task=`23ee456d-fd6b-48c2-9e03-932941e20344`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:32,573::task::595::TaskManager.Task::(_updateState) Task=`23ee456d-fd6b-48c2-9e03-932941e20344`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:32,573::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:32,574::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:32,575::task::990::TaskManager.Task::(_decref) Task=`23ee456d-fd6b-48c2-9e03-932941e20344`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:01:33,572::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-13::DEBUG::2014-04-11 22:01:33,590::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:33,597::task::595::TaskManager.Task::(_updateState) Task=`6f5d3752-608a-40c2-a826-bd93de918ae1`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:33,602::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:33,606::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:33,610::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:33,613::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:33,617::task::1185::TaskManager.Task::(prepare) Task=`6f5d3752-608a-40c2-a826-bd93de918ae1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:33,621::task::595::TaskManager.Task::(_updateState) Task=`6f5d3752-608a-40c2-a826-bd93de918ae1`::moving from state preparing -> state finished >Thread-44::DEBUG::2014-04-11 22:01:33,624::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000250837 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:33,625::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:33,630::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:33,632::task::990::TaskManager.Task::(_decref) Task=`6f5d3752-608a-40c2-a826-bd93de918ae1`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:34,647::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:34,649::task::595::TaskManager.Task::(_updateState) Task=`d538c307-bdcc-4d36-8f17-7c0d42120d3c`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:34,650::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:34,651::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:34,651::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:34,652::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:34,652::task::1185::TaskManager.Task::(prepare) Task=`d538c307-bdcc-4d36-8f17-7c0d42120d3c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:34,653::task::595::TaskManager.Task::(_updateState) Task=`d538c307-bdcc-4d36-8f17-7c0d42120d3c`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:34,654::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:34,654::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:34,655::task::990::TaskManager.Task::(_decref) Task=`d538c307-bdcc-4d36-8f17-7c0d42120d3c`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:35,662::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:35,665::task::595::TaskManager.Task::(_updateState) Task=`8bf581da-50e3-4be2-a580-57f6db88e596`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:35,666::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:35,667::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:35,667::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:35,669::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:35,671::task::1185::TaskManager.Task::(prepare) Task=`8bf581da-50e3-4be2-a580-57f6db88e596`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:35,671::task::595::TaskManager.Task::(_updateState) Task=`8bf581da-50e3-4be2-a580-57f6db88e596`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:35,671::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:35,673::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:35,673::task::990::TaskManager.Task::(_decref) Task=`8bf581da-50e3-4be2-a580-57f6db88e596`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:35,765::task::595::TaskManager.Task::(_updateState) Task=`3e15d61c-984d-44c5-90f3-7ebe7553d3dc`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:35,767::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-13::INFO::2014-04-11 22:01:35,769::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '64.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000273063', 'lastCheck': '3.7', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '25.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '42.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000250837', 'lastCheck': '2.1', 'valid': True}} >Thread-13::DEBUG::2014-04-11 22:01:35,769::task::1185::TaskManager.Task::(prepare) Task=`3e15d61c-984d-44c5-90f3-7ebe7553d3dc`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000554962', 'lastCheck': '64.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000273063', 'lastCheck': '3.7', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '25.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '42.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000250837', 'lastCheck': '2.1', 'valid': True}} >Thread-13::DEBUG::2014-04-11 22:01:35,771::task::595::TaskManager.Task::(_updateState) Task=`3e15d61c-984d-44c5-90f3-7ebe7553d3dc`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:35,771::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:35,772::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:35,773::task::990::TaskManager.Task::(_decref) Task=`3e15d61c-984d-44c5-90f3-7ebe7553d3dc`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:36,684::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:36,691::task::595::TaskManager.Task::(_updateState) Task=`e44f47d0-1f92-4fe2-899a-af8271c48949`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:36,696::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:36,700::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:36,703::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:36,708::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:36,709::task::1185::TaskManager.Task::(prepare) Task=`e44f47d0-1f92-4fe2-899a-af8271c48949`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:36,711::task::595::TaskManager.Task::(_updateState) Task=`e44f47d0-1f92-4fe2-899a-af8271c48949`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:36,719::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:36,721::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:36,726::task::990::TaskManager.Task::(_decref) Task=`e44f47d0-1f92-4fe2-899a-af8271c48949`::ref 0 aborting False >Thread-968::WARNING::2014-04-11 22:01:37,001::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-974::DEBUG::2014-04-11 22:01:37,125::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-974::DEBUG::2014-04-11 22:01:37,146::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-974::DEBUG::2014-04-11 22:01:37,155::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-974::DEBUG::2014-04-11 22:01:37,162::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-974::DEBUG::2014-04-11 22:01:37,175::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-974::DEBUG::2014-04-11 22:01:37,186::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-974::DEBUG::2014-04-11 22:01:37,191::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-974::DEBUG::2014-04-11 22:01:37,197::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-974::DEBUG::2014-04-11 22:01:37,235::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-981::WARNING::2014-04-11 22:01:37,379::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-977::ERROR::2014-04-11 22:01:37,384::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-31::DEBUG::2014-04-11 22:01:37,387::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-31::DEBUG::2014-04-11 22:01:37,388::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-31::DEBUG::2014-04-11 22:01:37,389::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-31::DEBUG::2014-04-11 22:01:37,391::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-31::DEBUG::2014-04-11 22:01:37,391::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-31::DEBUG::2014-04-11 22:01:37,392::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-31::DEBUG::2014-04-11 22:01:37,442::misc::772::SamplingMethod::(__call__) Returning last result >Thread-31::DEBUG::2014-04-11 22:01:37,448::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-988::WARNING::2014-04-11 22:01:37,463::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-988::DEBUG::2014-04-11 22:01:37,468::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-978::ERROR::2014-04-11 22:01:37,470::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-38::DEBUG::2014-04-11 22:01:37,472::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-31::DEBUG::2014-04-11 22:01:37,603::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-31::DEBUG::2014-04-11 22:01:37,604::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-31::DEBUG::2014-04-11 22:01:37,605::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-31::DEBUG::2014-04-11 22:01:37,606::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-31::DEBUG::2014-04-11 22:01:37,606::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-31::DEBUG::2014-04-11 22:01:37,606::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-31::DEBUG::2014-04-11 22:01:37,607::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-31::DEBUG::2014-04-11 22:01:37,608::misc::772::SamplingMethod::(__call__) Returning last result >Thread-38::DEBUG::2014-04-11 22:01:37,608::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-31::ERROR::2014-04-11 22:01:37,610::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-38::DEBUG::2014-04-11 22:01:37,611::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-31::ERROR::2014-04-11 22:01:37,611::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-38::DEBUG::2014-04-11 22:01:37,614::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-31::DEBUG::2014-04-11 22:01:37,615::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:37,617::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-38::DEBUG::2014-04-11 22:01:37,622::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-31::DEBUG::2014-04-11 22:01:37,637::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-38::DEBUG::2014-04-11 22:01:37,650::misc::772::SamplingMethod::(__call__) Returning last result >Thread-38::DEBUG::2014-04-11 22:01:37,651::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-38::DEBUG::2014-04-11 22:01:37,710::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-38::DEBUG::2014-04-11 22:01:37,712::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-13::DEBUG::2014-04-11 22:01:37,742::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:37,743::task::595::TaskManager.Task::(_updateState) Task=`b9bd0f68-848f-4986-a6c7-1b607add2b54`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:37,743::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:37,744::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:37,744::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:37,745::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:37,745::task::1185::TaskManager.Task::(prepare) Task=`b9bd0f68-848f-4986-a6c7-1b607add2b54`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:37,745::task::595::TaskManager.Task::(_updateState) Task=`b9bd0f68-848f-4986-a6c7-1b607add2b54`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:37,746::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:37,746::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:37,747::task::990::TaskManager.Task::(_decref) Task=`b9bd0f68-848f-4986-a6c7-1b607add2b54`::ref 0 aborting False >Thread-31::DEBUG::2014-04-11 22:01:37,944::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-31::WARNING::2014-04-11 22:01:37,946::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-31::DEBUG::2014-04-11 22:01:37,947::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:37,951::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:37,959::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:37,989::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:38,011::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:38,021::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:38,022::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:38,022::misc::772::SamplingMethod::(__call__) Returning last result >Thread-38::ERROR::2014-04-11 22:01:38,023::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-38::ERROR::2014-04-11 22:01:38,023::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-38::DEBUG::2014-04-11 22:01:38,024::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-38::DEBUG::2014-04-11 22:01:38,025::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-38::DEBUG::2014-04-11 22:01:38,306::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-38::WARNING::2014-04-11 22:01:38,308::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-38::DEBUG::2014-04-11 22:01:38,309::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-13::DEBUG::2014-04-11 22:01:38,753::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:38,754::task::595::TaskManager.Task::(_updateState) Task=`32e33d54-c63a-4281-aa05-6bda21433af4`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:38,755::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:38,755::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:38,756::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:38,757::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:38,757::task::1185::TaskManager.Task::(prepare) Task=`32e33d54-c63a-4281-aa05-6bda21433af4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:38,757::task::595::TaskManager.Task::(_updateState) Task=`32e33d54-c63a-4281-aa05-6bda21433af4`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:38,758::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:38,758::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:38,759::task::990::TaskManager.Task::(_decref) Task=`32e33d54-c63a-4281-aa05-6bda21433af4`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:39,764::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:39,765::task::595::TaskManager.Task::(_updateState) Task=`3fe5aae2-325b-4f68-bd80-abcd43752457`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:39,765::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:39,766::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:39,766::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:39,767::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:39,767::task::1185::TaskManager.Task::(prepare) Task=`3fe5aae2-325b-4f68-bd80-abcd43752457`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:39,768::task::595::TaskManager.Task::(_updateState) Task=`3fe5aae2-325b-4f68-bd80-abcd43752457`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:39,768::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:39,768::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:39,769::task::990::TaskManager.Task::(_decref) Task=`3fe5aae2-325b-4f68-bd80-abcd43752457`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:40,774::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:40,775::task::595::TaskManager.Task::(_updateState) Task=`0d3e91b9-f0a1-4d1d-a52d-a96678bceacf`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:40,776::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:40,777::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:40,777::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:40,778::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:40,779::task::1185::TaskManager.Task::(prepare) Task=`0d3e91b9-f0a1-4d1d-a52d-a96678bceacf`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:40,779::task::595::TaskManager.Task::(_updateState) Task=`0d3e91b9-f0a1-4d1d-a52d-a96678bceacf`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:40,780::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:40,781::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:40,781::task::990::TaskManager.Task::(_decref) Task=`0d3e91b9-f0a1-4d1d-a52d-a96678bceacf`::ref 0 aborting False >Thread-20::ERROR::2014-04-11 22:01:41,241::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-20::DEBUG::2014-04-11 22:01:41,242::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 changed its status to Invalid >Thread-1008::DEBUG::2014-04-11 22:01:41,243::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-20::DEBUG::2014-04-11 22:01:41,244::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-1008::DEBUG::2014-04-11 22:01:41,257::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-1008::DEBUG::2014-04-11 22:01:41,259::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_upgradePoolDomain` >Thread-1008::DEBUG::2014-04-11 22:01:41,261::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-13::DEBUG::2014-04-11 22:01:41,788::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:41,789::task::595::TaskManager.Task::(_updateState) Task=`1267a86e-0d04-4ace-9081-8abe16bc6833`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:41,790::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:41,790::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:41,790::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:41,791::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:41,791::task::1185::TaskManager.Task::(prepare) Task=`1267a86e-0d04-4ace-9081-8abe16bc6833`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:41,792::task::595::TaskManager.Task::(_updateState) Task=`1267a86e-0d04-4ace-9081-8abe16bc6833`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:41,792::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:41,792::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:41,793::task::990::TaskManager.Task::(_decref) Task=`1267a86e-0d04-4ace-9081-8abe16bc6833`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:01:42,085::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:01:42,104::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000265116 s, 1.4 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:42,798::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:42,814::task::595::TaskManager.Task::(_updateState) Task=`a9543da0-ec8e-4ba9-b6c7-a06f0bddd202`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:42,815::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:42,815::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:42,816::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:42,816::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:42,816::task::1185::TaskManager.Task::(prepare) Task=`a9543da0-ec8e-4ba9-b6c7-a06f0bddd202`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:42,817::task::595::TaskManager.Task::(_updateState) Task=`a9543da0-ec8e-4ba9-b6c7-a06f0bddd202`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:42,818::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:42,818::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:42,818::task::990::TaskManager.Task::(_decref) Task=`a9543da0-ec8e-4ba9-b6c7-a06f0bddd202`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:01:43,670::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:01:43,683::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000273416 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:01:43,823::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:43,824::task::595::TaskManager.Task::(_updateState) Task=`2493e9cb-7ab6-4f77-b13b-0e75fc1e23c6`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:43,826::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:43,826::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:43,826::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:43,827::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:43,827::task::1185::TaskManager.Task::(prepare) Task=`2493e9cb-7ab6-4f77-b13b-0e75fc1e23c6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:43,827::task::595::TaskManager.Task::(_updateState) Task=`2493e9cb-7ab6-4f77-b13b-0e75fc1e23c6`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:43,828::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:43,828::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:43,828::task::990::TaskManager.Task::(_decref) Task=`2493e9cb-7ab6-4f77-b13b-0e75fc1e23c6`::ref 0 aborting False >Thread-13::DEBUG::2014-04-11 22:01:44,833::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-13::DEBUG::2014-04-11 22:01:44,834::task::595::TaskManager.Task::(_updateState) Task=`9c65ec9b-75f4-427d-9581-d8d7e200a487`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:01:44,834::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-13::DEBUG::2014-04-11 22:01:44,834::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-13::DEBUG::2014-04-11 22:01:44,835::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-13::INFO::2014-04-11 22:01:44,835::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:44,835::task::1185::TaskManager.Task::(prepare) Task=`9c65ec9b-75f4-427d-9581-d8d7e200a487`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-13::DEBUG::2014-04-11 22:01:44,836::task::595::TaskManager.Task::(_updateState) Task=`9c65ec9b-75f4-427d-9581-d8d7e200a487`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:01:44,836::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:01:44,836::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:01:44,837::task::990::TaskManager.Task::(_decref) Task=`9c65ec9b-75f4-427d-9581-d8d7e200a487`::ref 0 aborting False >Thread-155::ERROR::2014-04-11 22:01:44,987::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect > self._mount.mount(self.options, self._vfsType) > File "/usr/share/vdsm/storage/mount.py", line 222, in mount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-155::ERROR::2014-04-11 22:01:44,990::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer > conObj.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect > return self._mountCon.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect > raise e >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-155::DEBUG::2014-04-11 22:01:44,990::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain} >Thread-155::INFO::2014-04-11 22:01:44,991::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}]} >Thread-155::DEBUG::2014-04-11 22:01:44,991::task::1185::TaskManager.Task::(prepare) Task=`49488c60-1e2d-446c-b2c8-d8ce81440468`::finished: {'statuslist': [{'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}]} >Thread-155::DEBUG::2014-04-11 22:01:44,992::task::595::TaskManager.Task::(_updateState) Task=`49488c60-1e2d-446c-b2c8-d8ce81440468`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:44,992::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:44,992::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:44,993::task::990::TaskManager.Task::(_decref) Task=`49488c60-1e2d-446c-b2c8-d8ce81440468`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:45,841::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:45,842::task::595::TaskManager.Task::(_updateState) Task=`bdce2879-b610-4150-bdba-eed36dd91f3e`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:45,842::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:45,843::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:45,843::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:45,843::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:45,844::task::1185::TaskManager.Task::(prepare) Task=`bdce2879-b610-4150-bdba-eed36dd91f3e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:45,844::task::595::TaskManager.Task::(_updateState) Task=`bdce2879-b610-4150-bdba-eed36dd91f3e`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:45,844::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:45,845::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:45,845::task::990::TaskManager.Task::(_decref) Task=`bdce2879-b610-4150-bdba-eed36dd91f3e`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:46,856::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:46,857::task::595::TaskManager.Task::(_updateState) Task=`f69a27b2-a38e-408e-a51d-c0c22f079da7`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:46,857::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:46,858::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:46,858::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:46,858::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:46,859::task::1185::TaskManager.Task::(prepare) Task=`f69a27b2-a38e-408e-a51d-c0c22f079da7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:46,859::task::595::TaskManager.Task::(_updateState) Task=`f69a27b2-a38e-408e-a51d-c0c22f079da7`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:46,859::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:46,860::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:46,860::task::990::TaskManager.Task::(_decref) Task=`f69a27b2-a38e-408e-a51d-c0c22f079da7`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:47,865::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:47,866::task::595::TaskManager.Task::(_updateState) Task=`a80c4a15-a7ff-4a9e-9ddd-476e07f136a8`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:47,866::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:47,866::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:47,867::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:47,867::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:47,867::task::1185::TaskManager.Task::(prepare) Task=`a80c4a15-a7ff-4a9e-9ddd-476e07f136a8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:47,867::task::595::TaskManager.Task::(_updateState) Task=`a80c4a15-a7ff-4a9e-9ddd-476e07f136a8`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:47,868::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:47,868::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:47,868::task::990::TaskManager.Task::(_decref) Task=`a80c4a15-a7ff-4a9e-9ddd-476e07f136a8`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:48,904::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:48,906::task::595::TaskManager.Task::(_updateState) Task=`540da1e6-f026-4bc1-8a03-e2ebc5343e8c`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:48,907::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:48,907::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:48,907::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:48,908::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:48,908::task::1185::TaskManager.Task::(prepare) Task=`540da1e6-f026-4bc1-8a03-e2ebc5343e8c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:48,908::task::595::TaskManager.Task::(_updateState) Task=`540da1e6-f026-4bc1-8a03-e2ebc5343e8c`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:48,908::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:48,909::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:48,909::task::990::TaskManager.Task::(_decref) Task=`540da1e6-f026-4bc1-8a03-e2ebc5343e8c`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:49,914::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:49,915::task::595::TaskManager.Task::(_updateState) Task=`36062093-f7ea-4107-be4d-1904abc4628a`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:49,915::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:49,915::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:49,916::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:49,916::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:49,916::task::1185::TaskManager.Task::(prepare) Task=`36062093-f7ea-4107-be4d-1904abc4628a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:49,917::task::595::TaskManager.Task::(_updateState) Task=`36062093-f7ea-4107-be4d-1904abc4628a`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:49,917::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:49,917::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:49,917::task::990::TaskManager.Task::(_decref) Task=`36062093-f7ea-4107-be4d-1904abc4628a`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:50,922::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:50,923::task::595::TaskManager.Task::(_updateState) Task=`bd63bc78-e5cc-4151-b220-5ab7f462c7b0`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:50,923::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:50,923::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:50,923::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:50,924::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:50,924::task::1185::TaskManager.Task::(prepare) Task=`bd63bc78-e5cc-4151-b220-5ab7f462c7b0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:50,924::task::595::TaskManager.Task::(_updateState) Task=`bd63bc78-e5cc-4151-b220-5ab7f462c7b0`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:50,925::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:50,925::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:50,925::task::990::TaskManager.Task::(_decref) Task=`bd63bc78-e5cc-4151-b220-5ab7f462c7b0`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:51,182::task::595::TaskManager.Task::(_updateState) Task=`e31f6fc3-d293-40b6-ab92-ccb8b32b385a`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:51,182::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-155::INFO::2014-04-11 22:01:51,183::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.9', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000265116', 'lastCheck': '9.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '79.5', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '41.4', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '79.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '58.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000273416', 'lastCheck': '7.5', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:01:51,183::task::1185::TaskManager.Task::(prepare) Task=`e31f6fc3-d293-40b6-ab92-ccb8b32b385a`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.9', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000265116', 'lastCheck': '9.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '79.5', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '41.4', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '79.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000485419', 'lastCheck': '58.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000273416', 'lastCheck': '7.5', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:01:51,184::task::595::TaskManager.Task::(_updateState) Task=`e31f6fc3-d293-40b6-ab92-ccb8b32b385a`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:51,184::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:51,184::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:51,185::task::990::TaskManager.Task::(_decref) Task=`e31f6fc3-d293-40b6-ab92-ccb8b32b385a`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:51,930::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:51,932::task::595::TaskManager.Task::(_updateState) Task=`8a4464d2-fb78-45c4-becb-2b9af2341346`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:51,932::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:51,933::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:51,933::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:51,934::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:51,934::task::1185::TaskManager.Task::(prepare) Task=`8a4464d2-fb78-45c4-becb-2b9af2341346`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:51,935::task::595::TaskManager.Task::(_updateState) Task=`8a4464d2-fb78-45c4-becb-2b9af2341346`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:51,935::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:51,935::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:51,936::task::990::TaskManager.Task::(_decref) Task=`8a4464d2-fb78-45c4-becb-2b9af2341346`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:01:52,137::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:01:52,152::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000293023 s, 1.3 MB/s\n'; <rc> = 0 >Thread-155::DEBUG::2014-04-11 22:01:52,941::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:52,942::task::595::TaskManager.Task::(_updateState) Task=`22c17f8c-b846-49fc-a1b6-8fe3fa940161`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:52,943::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:52,943::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:52,944::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:52,944::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:52,944::task::1185::TaskManager.Task::(prepare) Task=`22c17f8c-b846-49fc-a1b6-8fe3fa940161`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:52,944::task::595::TaskManager.Task::(_updateState) Task=`22c17f8c-b846-49fc-a1b6-8fe3fa940161`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:52,945::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:52,945::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:52,945::task::990::TaskManager.Task::(_decref) Task=`22c17f8c-b846-49fc-a1b6-8fe3fa940161`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:01:53,713::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:01:53,730::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000342916 s, 2.2 MB/s\n'; <rc> = 0 >Thread-155::DEBUG::2014-04-11 22:01:53,950::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:53,951::task::595::TaskManager.Task::(_updateState) Task=`2d48ccc0-a68d-4537-b082-b3a86cda3c24`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:53,952::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:53,952::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:53,953::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:53,953::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:53,953::task::1185::TaskManager.Task::(prepare) Task=`2d48ccc0-a68d-4537-b082-b3a86cda3c24`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:53,954::task::595::TaskManager.Task::(_updateState) Task=`2d48ccc0-a68d-4537-b082-b3a86cda3c24`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:53,955::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:53,955::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:53,955::task::990::TaskManager.Task::(_decref) Task=`2d48ccc0-a68d-4537-b082-b3a86cda3c24`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:54,960::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:54,961::task::595::TaskManager.Task::(_updateState) Task=`3b40b2cb-dff2-4746-9b70-c7064f8fdf59`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:54,961::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:54,961::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:54,962::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:54,962::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:54,962::task::1185::TaskManager.Task::(prepare) Task=`3b40b2cb-dff2-4746-9b70-c7064f8fdf59`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:54,963::task::595::TaskManager.Task::(_updateState) Task=`3b40b2cb-dff2-4746-9b70-c7064f8fdf59`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:54,963::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:54,963::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:54,964::task::990::TaskManager.Task::(_decref) Task=`3b40b2cb-dff2-4746-9b70-c7064f8fdf59`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:55,969::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:55,969::task::595::TaskManager.Task::(_updateState) Task=`804e8e1b-97ac-493d-a1f3-5e53776eb71f`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:55,970::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:55,970::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:55,970::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:55,971::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:55,971::task::1185::TaskManager.Task::(prepare) Task=`804e8e1b-97ac-493d-a1f3-5e53776eb71f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:55,971::task::595::TaskManager.Task::(_updateState) Task=`804e8e1b-97ac-493d-a1f3-5e53776eb71f`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:55,972::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:55,972::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:55,972::task::990::TaskManager.Task::(_decref) Task=`804e8e1b-97ac-493d-a1f3-5e53776eb71f`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:56,977::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:56,979::task::595::TaskManager.Task::(_updateState) Task=`957aa3e4-1587-4adc-88d5-a8cd05be08cd`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:56,979::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:56,980::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:56,980::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:56,981::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:56,981::task::1185::TaskManager.Task::(prepare) Task=`957aa3e4-1587-4adc-88d5-a8cd05be08cd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:56,981::task::595::TaskManager.Task::(_updateState) Task=`957aa3e4-1587-4adc-88d5-a8cd05be08cd`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:56,982::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:56,982::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:56,982::task::990::TaskManager.Task::(_decref) Task=`957aa3e4-1587-4adc-88d5-a8cd05be08cd`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:57,987::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:57,989::task::595::TaskManager.Task::(_updateState) Task=`77016bb1-dc25-4510-86a8-855cf9a1d207`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:57,989::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:57,989::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:57,990::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:57,990::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:57,990::task::1185::TaskManager.Task::(prepare) Task=`77016bb1-dc25-4510-86a8-855cf9a1d207`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:57,991::task::595::TaskManager.Task::(_updateState) Task=`77016bb1-dc25-4510-86a8-855cf9a1d207`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:57,991::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:57,991::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:57,992::task::990::TaskManager.Task::(_decref) Task=`77016bb1-dc25-4510-86a8-855cf9a1d207`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:01:58,996::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:01:58,997::task::595::TaskManager.Task::(_updateState) Task=`fc2782e8-4bd7-4d80-8fc4-cd3ab655ead1`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:01:58,997::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:01:58,998::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:01:58,998::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:01:58,998::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:58,999::task::1185::TaskManager.Task::(prepare) Task=`fc2782e8-4bd7-4d80-8fc4-cd3ab655ead1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:01:58,999::task::595::TaskManager.Task::(_updateState) Task=`fc2782e8-4bd7-4d80-8fc4-cd3ab655ead1`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:01:58,999::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:01:59,000::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:01:59,000::task::990::TaskManager.Task::(_decref) Task=`fc2782e8-4bd7-4d80-8fc4-cd3ab655ead1`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:00,004::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:00,005::task::595::TaskManager.Task::(_updateState) Task=`90aca4f8-9bb3-4b9d-bc48-cde3c189a242`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:00,005::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:00,006::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:00,006::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:00,006::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:00,007::task::1185::TaskManager.Task::(prepare) Task=`90aca4f8-9bb3-4b9d-bc48-cde3c189a242`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:00,007::task::595::TaskManager.Task::(_updateState) Task=`90aca4f8-9bb3-4b9d-bc48-cde3c189a242`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:00,007::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:00,008::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:00,008::task::990::TaskManager.Task::(_decref) Task=`90aca4f8-9bb3-4b9d-bc48-cde3c189a242`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:01,013::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:01,013::task::595::TaskManager.Task::(_updateState) Task=`c6da1f71-1d0d-44d5-9e9e-b51db9c21ff4`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:01,014::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:01,014::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:01,015::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:01,015::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:01,015::task::1185::TaskManager.Task::(prepare) Task=`c6da1f71-1d0d-44d5-9e9e-b51db9c21ff4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:01,016::task::595::TaskManager.Task::(_updateState) Task=`c6da1f71-1d0d-44d5-9e9e-b51db9c21ff4`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:01,016::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:01,016::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:01,017::task::990::TaskManager.Task::(_decref) Task=`c6da1f71-1d0d-44d5-9e9e-b51db9c21ff4`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:02,025::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:02,030::task::595::TaskManager.Task::(_updateState) Task=`715fa805-cb26-4bff-9c08-36f6713c8700`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:02,034::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:02,036::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:02,038::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:02,041::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:02,043::task::1185::TaskManager.Task::(prepare) Task=`715fa805-cb26-4bff-9c08-36f6713c8700`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:02,046::task::595::TaskManager.Task::(_updateState) Task=`715fa805-cb26-4bff-9c08-36f6713c8700`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:02,048::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:02,050::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:02,053::task::990::TaskManager.Task::(_decref) Task=`715fa805-cb26-4bff-9c08-36f6713c8700`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:02:02,357::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:02:02,400::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000276526 s, 1.3 MB/s\n'; <rc> = 0 >Thread-43::ERROR::2014-04-11 22:02:02,979::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain a033c2ac-0d01-490c-9552-99ca53d6a64a monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-43::DEBUG::2014-04-11 22:02:02,981::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain a033c2ac-0d01-490c-9552-99ca53d6a64a changed its status to Invalid >Thread-1011::DEBUG::2014-04-11 22:02:02,983::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-1011::DEBUG::2014-04-11 22:02:02,986::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-1011::DEBUG::2014-04-11 22:02:02,988::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_upgradePoolDomain` >Thread-1011::DEBUG::2014-04-11 22:02:02,989::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-155::DEBUG::2014-04-11 22:02:03,065::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:03,066::task::595::TaskManager.Task::(_updateState) Task=`3677bb8f-f130-46ac-a942-7771a4312d1d`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:03,066::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:03,067::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:03,067::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:03,068::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:03,068::task::1185::TaskManager.Task::(prepare) Task=`3677bb8f-f130-46ac-a942-7771a4312d1d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:03,069::task::595::TaskManager.Task::(_updateState) Task=`3677bb8f-f130-46ac-a942-7771a4312d1d`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:03,069::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:03,070::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:03,070::task::990::TaskManager.Task::(_decref) Task=`3677bb8f-f130-46ac-a942-7771a4312d1d`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:02:03,766::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:02:03,781::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000265086 s, 2.8 MB/s\n'; <rc> = 0 >Thread-155::DEBUG::2014-04-11 22:02:04,075::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:04,076::task::595::TaskManager.Task::(_updateState) Task=`908b8349-c627-48c7-9e8b-75cd7cb202ca`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:04,077::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:04,077::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:04,077::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:04,078::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:04,078::task::1185::TaskManager.Task::(prepare) Task=`908b8349-c627-48c7-9e8b-75cd7cb202ca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:04,078::task::595::TaskManager.Task::(_updateState) Task=`908b8349-c627-48c7-9e8b-75cd7cb202ca`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:04,079::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:04,079::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:04,079::task::990::TaskManager.Task::(_decref) Task=`908b8349-c627-48c7-9e8b-75cd7cb202ca`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:05,084::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:05,085::task::595::TaskManager.Task::(_updateState) Task=`441ae0b9-ecc4-432f-92e9-e1c45f501024`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:05,085::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:05,085::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:05,086::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:05,086::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:05,086::task::1185::TaskManager.Task::(prepare) Task=`441ae0b9-ecc4-432f-92e9-e1c45f501024`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:05,087::task::595::TaskManager.Task::(_updateState) Task=`441ae0b9-ecc4-432f-92e9-e1c45f501024`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:05,087::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:05,087::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:05,088::task::990::TaskManager.Task::(_decref) Task=`441ae0b9-ecc4-432f-92e9-e1c45f501024`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:06,092::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:06,093::task::595::TaskManager.Task::(_updateState) Task=`83e96f57-9f41-4329-86f9-44f1fa06c7f7`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:06,093::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:06,094::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:06,094::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:06,094::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:06,094::task::1185::TaskManager.Task::(prepare) Task=`83e96f57-9f41-4329-86f9-44f1fa06c7f7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:06,095::task::595::TaskManager.Task::(_updateState) Task=`83e96f57-9f41-4329-86f9-44f1fa06c7f7`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:06,095::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:06,095::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:06,096::task::990::TaskManager.Task::(_decref) Task=`83e96f57-9f41-4329-86f9-44f1fa06c7f7`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:06,531::task::595::TaskManager.Task::(_updateState) Task=`591d1b40-c01c-47f1-9a0c-a988a168f689`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:06,532::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-155::INFO::2014-04-11 22:02:06,532::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '25.3', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000276526', 'lastCheck': '4.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '94.8', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '56.7', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '94.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.6', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000265086', 'lastCheck': '2.7', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:02:06,533::task::1185::TaskManager.Task::(prepare) Task=`591d1b40-c01c-47f1-9a0c-a988a168f689`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '25.3', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000276526', 'lastCheck': '4.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '94.8', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000546938', 'lastCheck': '56.7', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '94.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.6', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000265086', 'lastCheck': '2.7', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:02:06,533::task::595::TaskManager.Task::(_updateState) Task=`591d1b40-c01c-47f1-9a0c-a988a168f689`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:06,533::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:06,534::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:06,534::task::990::TaskManager.Task::(_decref) Task=`591d1b40-c01c-47f1-9a0c-a988a168f689`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:07,100::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:07,101::task::595::TaskManager.Task::(_updateState) Task=`354ddccf-f915-452e-a434-08cf6e65c08b`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:07,102::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:07,102::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:07,103::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:07,103::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:07,103::task::1185::TaskManager.Task::(prepare) Task=`354ddccf-f915-452e-a434-08cf6e65c08b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:07,104::task::595::TaskManager.Task::(_updateState) Task=`354ddccf-f915-452e-a434-08cf6e65c08b`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:07,104::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:07,105::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:07,105::task::990::TaskManager.Task::(_decref) Task=`354ddccf-f915-452e-a434-08cf6e65c08b`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:08,109::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:08,110::task::595::TaskManager.Task::(_updateState) Task=`e67c32db-67d3-43df-b177-05338f8c05b4`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:08,110::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:08,111::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:08,111::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:08,111::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:08,112::task::1185::TaskManager.Task::(prepare) Task=`e67c32db-67d3-43df-b177-05338f8c05b4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:08,112::task::595::TaskManager.Task::(_updateState) Task=`e67c32db-67d3-43df-b177-05338f8c05b4`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:08,112::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:08,113::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:08,113::task::990::TaskManager.Task::(_decref) Task=`e67c32db-67d3-43df-b177-05338f8c05b4`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:09,117::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:09,118::task::595::TaskManager.Task::(_updateState) Task=`ce387f57-8f67-4e2b-9e6d-3a6c1b2e61ab`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:09,118::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:09,118::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:09,119::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:09,119::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:09,119::task::1185::TaskManager.Task::(prepare) Task=`ce387f57-8f67-4e2b-9e6d-3a6c1b2e61ab`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:09,120::task::595::TaskManager.Task::(_updateState) Task=`ce387f57-8f67-4e2b-9e6d-3a6c1b2e61ab`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:09,120::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:09,120::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:09,121::task::990::TaskManager.Task::(_decref) Task=`ce387f57-8f67-4e2b-9e6d-3a6c1b2e61ab`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:10,124::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:10,125::task::595::TaskManager.Task::(_updateState) Task=`536a39ae-18ed-4ede-8ef5-e25e9b5f1f34`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:10,125::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:10,126::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:10,126::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:10,126::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:10,127::task::1185::TaskManager.Task::(prepare) Task=`536a39ae-18ed-4ede-8ef5-e25e9b5f1f34`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:10,127::task::595::TaskManager.Task::(_updateState) Task=`536a39ae-18ed-4ede-8ef5-e25e9b5f1f34`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:10,127::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:10,127::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:10,128::task::990::TaskManager.Task::(_decref) Task=`536a39ae-18ed-4ede-8ef5-e25e9b5f1f34`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:11,132::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:11,133::task::595::TaskManager.Task::(_updateState) Task=`a2dc7b58-a809-4d45-b29d-5d9cf32cca45`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:11,133::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:11,134::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:11,134::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:11,134::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:11,134::task::1185::TaskManager.Task::(prepare) Task=`a2dc7b58-a809-4d45-b29d-5d9cf32cca45`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:11,135::task::595::TaskManager.Task::(_updateState) Task=`a2dc7b58-a809-4d45-b29d-5d9cf32cca45`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:11,135::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:11,135::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:11,136::task::990::TaskManager.Task::(_decref) Task=`a2dc7b58-a809-4d45-b29d-5d9cf32cca45`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:12,141::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:12,142::task::595::TaskManager.Task::(_updateState) Task=`906ac308-0e79-42ef-9893-3c746eb4eca5`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:12,143::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:12,143::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:12,143::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:12,144::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:12,144::task::1185::TaskManager.Task::(prepare) Task=`906ac308-0e79-42ef-9893-3c746eb4eca5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:12,145::task::595::TaskManager.Task::(_updateState) Task=`906ac308-0e79-42ef-9893-3c746eb4eca5`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:12,145::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:12,145::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:12,146::task::990::TaskManager.Task::(_decref) Task=`906ac308-0e79-42ef-9893-3c746eb4eca5`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:02:12,442::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:02:12,457::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000302115 s, 1.2 MB/s\n'; <rc> = 0 >Thread-155::DEBUG::2014-04-11 22:02:13,151::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:13,153::task::595::TaskManager.Task::(_updateState) Task=`4cb856a7-aed7-4666-bf8d-848227c5cd12`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:13,154::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:13,154::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:13,155::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:13,155::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:13,155::task::1185::TaskManager.Task::(prepare) Task=`4cb856a7-aed7-4666-bf8d-848227c5cd12`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:13,156::task::595::TaskManager.Task::(_updateState) Task=`4cb856a7-aed7-4666-bf8d-848227c5cd12`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:13,156::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:13,156::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:13,158::task::990::TaskManager.Task::(_decref) Task=`4cb856a7-aed7-4666-bf8d-848227c5cd12`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:02:13,810::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:02:13,825::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000279867 s, 2.7 MB/s\n'; <rc> = 0 >Thread-155::DEBUG::2014-04-11 22:02:14,163::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:14,165::task::595::TaskManager.Task::(_updateState) Task=`4e4714b7-847a-4c22-952e-55db0958b628`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:14,166::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:14,166::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:14,166::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:14,167::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:14,168::task::1185::TaskManager.Task::(prepare) Task=`4e4714b7-847a-4c22-952e-55db0958b628`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:14,168::task::595::TaskManager.Task::(_updateState) Task=`4e4714b7-847a-4c22-952e-55db0958b628`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:14,168::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:14,169::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:14,169::task::990::TaskManager.Task::(_decref) Task=`4e4714b7-847a-4c22-952e-55db0958b628`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:15,174::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:15,175::task::595::TaskManager.Task::(_updateState) Task=`e89b9630-ac3d-4213-9203-197915d7d755`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:15,175::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:15,176::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:15,176::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:15,176::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:15,177::task::1185::TaskManager.Task::(prepare) Task=`e89b9630-ac3d-4213-9203-197915d7d755`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:15,177::task::595::TaskManager.Task::(_updateState) Task=`e89b9630-ac3d-4213-9203-197915d7d755`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:15,177::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:15,178::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:15,178::task::990::TaskManager.Task::(_decref) Task=`e89b9630-ac3d-4213-9203-197915d7d755`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:16,183::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:16,184::task::595::TaskManager.Task::(_updateState) Task=`a6ac205f-e587-40d0-9bbd-d8ce41032c7b`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:16,184::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:16,185::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:16,185::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:16,185::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:16,186::task::1185::TaskManager.Task::(prepare) Task=`a6ac205f-e587-40d0-9bbd-d8ce41032c7b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:16,186::task::595::TaskManager.Task::(_updateState) Task=`a6ac205f-e587-40d0-9bbd-d8ce41032c7b`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:16,186::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:16,187::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:16,187::task::990::TaskManager.Task::(_decref) Task=`a6ac205f-e587-40d0-9bbd-d8ce41032c7b`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:17,192::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:17,194::task::595::TaskManager.Task::(_updateState) Task=`7a64a7b0-dcf8-42cc-8bf4-8fee283002f5`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:17,194::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:17,195::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:17,196::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:17,197::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:17,197::task::1185::TaskManager.Task::(prepare) Task=`7a64a7b0-dcf8-42cc-8bf4-8fee283002f5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:17,197::task::595::TaskManager.Task::(_updateState) Task=`7a64a7b0-dcf8-42cc-8bf4-8fee283002f5`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:17,198::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:17,198::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:17,199::task::990::TaskManager.Task::(_decref) Task=`7a64a7b0-dcf8-42cc-8bf4-8fee283002f5`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:18,204::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:18,205::task::595::TaskManager.Task::(_updateState) Task=`750c5398-819d-4eac-85d9-341efddbbef9`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:18,205::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:18,206::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:18,206::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:18,207::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:18,207::task::1185::TaskManager.Task::(prepare) Task=`750c5398-819d-4eac-85d9-341efddbbef9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:18,208::task::595::TaskManager.Task::(_updateState) Task=`750c5398-819d-4eac-85d9-341efddbbef9`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:18,208::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:18,209::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:18,209::task::990::TaskManager.Task::(_decref) Task=`750c5398-819d-4eac-85d9-341efddbbef9`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:19,219::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:19,225::task::595::TaskManager.Task::(_updateState) Task=`24a2f822-d893-48b7-9bf6-3d654aefc01b`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:19,230::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:19,233::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:19,237::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:19,239::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:19,241::task::1185::TaskManager.Task::(prepare) Task=`24a2f822-d893-48b7-9bf6-3d654aefc01b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:19,243::task::595::TaskManager.Task::(_updateState) Task=`24a2f822-d893-48b7-9bf6-3d654aefc01b`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:19,245::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:19,248::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:19,252::task::990::TaskManager.Task::(_decref) Task=`24a2f822-d893-48b7-9bf6-3d654aefc01b`::ref 0 aborting False >MainThread::DEBUG::2014-04-11 22:02:19,685::vdsm::55::vds::(sigtermHandler) Received signal 15 >MainThread::INFO::2014-04-11 22:02:19,806::vmChannels::183::vds::(stop) VM channels listener was stopped. >MainThread::INFO::2014-04-11 22:02:19,808::momIF::84::MOM::(stop) Shutting down MOM >MainThread::DEBUG::2014-04-11 22:02:19,810::task::595::TaskManager.Task::(_updateState) Task=`9b63cabe-d01c-48bb-9ebf-4e724b3add44`::moving from state init -> state preparing >MainThread::INFO::2014-04-11 22:02:19,810::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) >Thread-11::DEBUG::2014-04-11 22:02:19,812::storageServer::698::ConnectionMonitor::(_monitorConnections) Monitoring stopped >Thread-25::ERROR::2014-04-11 22:02:19,833::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-25::DEBUG::2014-04-11 22:02:19,836::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain b7663d70-e658-41fa-b9f0-8da83c9eddce changed its status to Invalid >Thread-1014::DEBUG::2014-04-11 22:02:19,846::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-1014::DEBUG::2014-04-11 22:02:19,851::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-1014::DEBUG::2014-04-11 22:02:19,856::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_upgradePoolDomain` >Thread-1014::DEBUG::2014-04-11 22:02:19,861::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >VM Channels Listener::INFO::2014-04-11 22:02:19,899::vmChannels::178::vds::(run) VM channels listener thread has ended. >Thread-155::DEBUG::2014-04-11 22:02:20,268::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:20,273::task::595::TaskManager.Task::(_updateState) Task=`25cada56-2c55-4c65-b7d8-ab8c102cfe13`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:20,276::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:20,279::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:20,285::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:20,287::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:20,289::task::1185::TaskManager.Task::(prepare) Task=`25cada56-2c55-4c65-b7d8-ab8c102cfe13`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:20,291::task::595::TaskManager.Task::(_updateState) Task=`25cada56-2c55-4c65-b7d8-ab8c102cfe13`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:20,295::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:20,301::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:20,304::task::990::TaskManager.Task::(_decref) Task=`25cada56-2c55-4c65-b7d8-ab8c102cfe13`::ref 0 aborting False >MainThread::DEBUG::2014-04-11 22:02:20,687::vdsm::55::vds::(sigtermHandler) Received signal 15 >MainThread::DEBUG::2014-04-11 22:02:20,687::clientIF::207::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently >Thread-155::DEBUG::2014-04-11 22:02:21,322::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:21,325::task::595::TaskManager.Task::(_updateState) Task=`d59aa97a-fd80-4165-b867-5ef2cb5cdc44`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:21,328::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:21,332::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:21,335::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:21,336::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:21,338::task::1185::TaskManager.Task::(prepare) Task=`d59aa97a-fd80-4165-b867-5ef2cb5cdc44`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:21,339::task::595::TaskManager.Task::(_updateState) Task=`d59aa97a-fd80-4165-b867-5ef2cb5cdc44`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:21,344::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:21,346::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:21,348::task::990::TaskManager.Task::(_decref) Task=`d59aa97a-fd80-4165-b867-5ef2cb5cdc44`::ref 0 aborting False >MainThread::DEBUG::2014-04-11 22:02:21,688::vdsm::55::vds::(sigtermHandler) Received signal 15 >MainThread::DEBUG::2014-04-11 22:02:21,688::clientIF::207::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently >Thread-155::DEBUG::2014-04-11 22:02:21,915::task::595::TaskManager.Task::(_updateState) Task=`569c8eb1-816a-48c0-8cd7-ab185821b22f`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:21,919::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-155::INFO::2014-04-11 22:02:21,925::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '40.7', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000302115', 'lastCheck': '9.5', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '110.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.1', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '110.3', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.9', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000279867', 'lastCheck': '8.1', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:02:21,927::task::1185::TaskManager.Task::(prepare) Task=`569c8eb1-816a-48c0-8cd7-ab185821b22f`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '40.7', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000302115', 'lastCheck': '9.5', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '110.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.1', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '110.3', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.9', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000279867', 'lastCheck': '8.1', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:02:21,929::task::595::TaskManager.Task::(_updateState) Task=`569c8eb1-816a-48c0-8cd7-ab185821b22f`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:21,933::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:21,935::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:21,937::task::990::TaskManager.Task::(_decref) Task=`569c8eb1-816a-48c0-8cd7-ab185821b22f`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:22,364::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:22,367::task::595::TaskManager.Task::(_updateState) Task=`7299981d-759d-4e2b-980c-5e4d519f0a3e`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:22,369::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:22,373::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:22,378::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:22,380::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:22,381::task::1185::TaskManager.Task::(prepare) Task=`7299981d-759d-4e2b-980c-5e4d519f0a3e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:22,383::task::595::TaskManager.Task::(_updateState) Task=`7299981d-759d-4e2b-980c-5e4d519f0a3e`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:22,390::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:22,393::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:22,395::task::990::TaskManager.Task::(_decref) Task=`7299981d-759d-4e2b-980c-5e4d519f0a3e`::ref 0 aborting False >MainThread::DEBUG::2014-04-11 22:02:22,700::vdsm::55::vds::(sigtermHandler) Received signal 15 >MainThread::DEBUG::2014-04-11 22:02:22,703::clientIF::207::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently >Thread-22::DEBUG::2014-04-11 22:02:22,738::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:02:22,775::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000207691 s, 1.8 MB/s\n'; <rc> = 0 >MainThread::INFO::2014-04-11 22:02:23,068::domainMonitor::125::Storage.DomainMonitor::(close) Stopping domain monitors >MainThread::INFO::2014-04-11 22:02:23,068::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-155::DEBUG::2014-04-11 22:02:23,407::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:23,408::task::595::TaskManager.Task::(_updateState) Task=`bdd239f1-2226-4b26-a150-333d6e9e367b`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:23,408::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:23,409::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:23,409::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:23,409::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:23,410::task::1185::TaskManager.Task::(prepare) Task=`bdd239f1-2226-4b26-a150-333d6e9e367b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:23,410::task::595::TaskManager.Task::(_updateState) Task=`bdd239f1-2226-4b26-a150-333d6e9e367b`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:23,410::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:23,410::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:23,411::task::990::TaskManager.Task::(_decref) Task=`bdd239f1-2226-4b26-a150-333d6e9e367b`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:02:23,861::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:02:23,878::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00024157 s, 3.1 MB/s\n'; <rc> = 0 >Thread-155::DEBUG::2014-04-11 22:02:24,416::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:24,417::task::595::TaskManager.Task::(_updateState) Task=`cbf02b2e-9801-44a9-a70d-c3730f8c5524`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:24,418::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:24,418::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:24,418::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:24,419::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:24,419::task::1185::TaskManager.Task::(prepare) Task=`cbf02b2e-9801-44a9-a70d-c3730f8c5524`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:24,420::task::595::TaskManager.Task::(_updateState) Task=`cbf02b2e-9801-44a9-a70d-c3730f8c5524`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:24,420::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:24,420::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:24,421::task::990::TaskManager.Task::(_decref) Task=`cbf02b2e-9801-44a9-a70d-c3730f8c5524`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:25,426::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:25,428::task::595::TaskManager.Task::(_updateState) Task=`2b5d84d9-ca57-4fe0-adef-957b67d0e76b`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:25,428::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:25,429::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:25,429::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:25,430::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:25,430::task::1185::TaskManager.Task::(prepare) Task=`2b5d84d9-ca57-4fe0-adef-957b67d0e76b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:25,431::task::595::TaskManager.Task::(_updateState) Task=`2b5d84d9-ca57-4fe0-adef-957b67d0e76b`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:25,431::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:25,431::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:25,432::task::990::TaskManager.Task::(_decref) Task=`2b5d84d9-ca57-4fe0-adef-957b67d0e76b`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:26,436::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:26,437::task::595::TaskManager.Task::(_updateState) Task=`03bf9766-f4c0-4eea-84b8-b8154ae294fb`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:26,437::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:26,438::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:26,438::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:26,438::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:26,439::task::1185::TaskManager.Task::(prepare) Task=`03bf9766-f4c0-4eea-84b8-b8154ae294fb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:26,439::task::595::TaskManager.Task::(_updateState) Task=`03bf9766-f4c0-4eea-84b8-b8154ae294fb`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:26,440::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:26,440::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:26,440::task::990::TaskManager.Task::(_decref) Task=`03bf9766-f4c0-4eea-84b8-b8154ae294fb`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:27,447::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:27,448::task::595::TaskManager.Task::(_updateState) Task=`5e466eb1-1d12-421d-bc56-30e286d026ce`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:27,448::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:27,449::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:27,449::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:27,449::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:27,450::task::1185::TaskManager.Task::(prepare) Task=`5e466eb1-1d12-421d-bc56-30e286d026ce`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:27,450::task::595::TaskManager.Task::(_updateState) Task=`5e466eb1-1d12-421d-bc56-30e286d026ce`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:27,450::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:27,451::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:27,452::task::990::TaskManager.Task::(_decref) Task=`5e466eb1-1d12-421d-bc56-30e286d026ce`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:28,456::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:28,457::task::595::TaskManager.Task::(_updateState) Task=`552ee523-3f64-402c-bd96-651e7f1c67be`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:28,458::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:28,458::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:28,459::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:28,459::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:28,460::task::1185::TaskManager.Task::(prepare) Task=`552ee523-3f64-402c-bd96-651e7f1c67be`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:28,460::task::595::TaskManager.Task::(_updateState) Task=`552ee523-3f64-402c-bd96-651e7f1c67be`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:28,461::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:28,461::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:28,461::task::990::TaskManager.Task::(_decref) Task=`552ee523-3f64-402c-bd96-651e7f1c67be`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:29,467::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:29,468::task::595::TaskManager.Task::(_updateState) Task=`cda0bb55-7abd-4454-8788-99fdfd784b02`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:29,468::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:29,469::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:29,469::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:29,469::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:29,471::task::1185::TaskManager.Task::(prepare) Task=`cda0bb55-7abd-4454-8788-99fdfd784b02`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:29,472::task::595::TaskManager.Task::(_updateState) Task=`cda0bb55-7abd-4454-8788-99fdfd784b02`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:29,472::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:29,472::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:29,473::task::990::TaskManager.Task::(_decref) Task=`cda0bb55-7abd-4454-8788-99fdfd784b02`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:30,480::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:30,482::task::595::TaskManager.Task::(_updateState) Task=`443f4517-e836-485e-94d3-e52c8c9573a5`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:30,482::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:30,482::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:30,483::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:30,483::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:30,484::task::1185::TaskManager.Task::(prepare) Task=`443f4517-e836-485e-94d3-e52c8c9573a5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:30,485::task::595::TaskManager.Task::(_updateState) Task=`443f4517-e836-485e-94d3-e52c8c9573a5`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:30,485::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:30,486::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:30,486::task::990::TaskManager.Task::(_decref) Task=`443f4517-e836-485e-94d3-e52c8c9573a5`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:31,492::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:31,493::task::595::TaskManager.Task::(_updateState) Task=`2c363f81-8294-4e9a-ac7b-a7cf65a58bd8`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:31,493::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:31,493::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:31,494::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:31,494::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:31,495::task::1185::TaskManager.Task::(prepare) Task=`2c363f81-8294-4e9a-ac7b-a7cf65a58bd8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:31,496::task::595::TaskManager.Task::(_updateState) Task=`2c363f81-8294-4e9a-ac7b-a7cf65a58bd8`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:31,497::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:31,497::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:31,498::task::990::TaskManager.Task::(_decref) Task=`2c363f81-8294-4e9a-ac7b-a7cf65a58bd8`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:32,503::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:32,505::task::595::TaskManager.Task::(_updateState) Task=`4775c86d-484e-4e81-9ee9-b2c69b3c7b5e`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:32,506::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:32,506::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:32,507::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:32,507::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:32,507::task::1185::TaskManager.Task::(prepare) Task=`4775c86d-484e-4e81-9ee9-b2c69b3c7b5e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:32,508::task::595::TaskManager.Task::(_updateState) Task=`4775c86d-484e-4e81-9ee9-b2c69b3c7b5e`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:32,508::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:32,509::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:32,509::task::990::TaskManager.Task::(_decref) Task=`4775c86d-484e-4e81-9ee9-b2c69b3c7b5e`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:02:32,832::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:02:32,847::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000268373 s, 1.4 MB/s\n'; <rc> = 0 >Thread-155::DEBUG::2014-04-11 22:02:33,517::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:33,520::task::595::TaskManager.Task::(_updateState) Task=`ec0f6158-d086-4957-affe-1552e341c7b6`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:33,521::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:33,522::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:33,522::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:33,522::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:33,526::task::1185::TaskManager.Task::(prepare) Task=`ec0f6158-d086-4957-affe-1552e341c7b6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:33,526::task::595::TaskManager.Task::(_updateState) Task=`ec0f6158-d086-4957-affe-1552e341c7b6`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:33,526::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:33,527::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:33,527::task::990::TaskManager.Task::(_decref) Task=`ec0f6158-d086-4957-affe-1552e341c7b6`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:02:33,937::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:02:33,951::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000257954 s, 2.9 MB/s\n'; <rc> = 0 >Thread-155::DEBUG::2014-04-11 22:02:34,536::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:34,537::task::595::TaskManager.Task::(_updateState) Task=`fd0d552f-a7ff-451d-926a-56fc9347a480`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:34,540::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:34,541::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:34,541::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:34,542::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:34,544::task::1185::TaskManager.Task::(prepare) Task=`fd0d552f-a7ff-451d-926a-56fc9347a480`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:34,544::task::595::TaskManager.Task::(_updateState) Task=`fd0d552f-a7ff-451d-926a-56fc9347a480`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:34,545::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:34,545::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:34,545::task::990::TaskManager.Task::(_decref) Task=`fd0d552f-a7ff-451d-926a-56fc9347a480`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:35,556::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:35,560::task::595::TaskManager.Task::(_updateState) Task=`03fe49be-bc7c-410d-9651-d0ccba62c999`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:35,563::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:35,565::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:35,567::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:35,569::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:35,571::task::1185::TaskManager.Task::(prepare) Task=`03fe49be-bc7c-410d-9651-d0ccba62c999`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:35,572::task::595::TaskManager.Task::(_updateState) Task=`03fe49be-bc7c-410d-9651-d0ccba62c999`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:35,574::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:35,575::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:35,578::task::990::TaskManager.Task::(_decref) Task=`03fe49be-bc7c-410d-9651-d0ccba62c999`::ref 0 aborting False >Thread-155::DEBUG::2014-04-11 22:02:36,600::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [16bbbbcc] >Thread-155::DEBUG::2014-04-11 22:02:36,615::task::595::TaskManager.Task::(_updateState) Task=`7959ef3d-5634-447f-8b77-06045d526aea`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:36,624::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9f4a86db-7348-40cf-aa72-ebd91cba08c0', spUUID=None, options=None) >Thread-155::DEBUG::2014-04-11 22:02:36,633::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9f4a86db-7348-40cf-aa72-ebd91cba08c0 >Thread-155::DEBUG::2014-04-11 22:02:36,642::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'} >Thread-155::INFO::2014-04-11 22:02:36,652::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:36,661::task::1185::TaskManager.Task::(prepare) Task=`7959ef3d-5634-447f-8b77-06045d526aea`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9f4a86db-7348-40cf-aa72-ebd91cba08c0'}} >Thread-155::DEBUG::2014-04-11 22:02:36,670::task::595::TaskManager.Task::(_updateState) Task=`7959ef3d-5634-447f-8b77-06045d526aea`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:36,680::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:36,687::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:36,696::task::990::TaskManager.Task::(_decref) Task=`7959ef3d-5634-447f-8b77-06045d526aea`::ref 0 aborting False >9f4a86db-7348-40cf-aa72-ebd91cba08c0::ERROR::2014-04-11 22:02:37,158::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1410, in checkBackupDomain > if dom.isBackup(): > File "/usr/share/vdsm/storage/sd.py", line 788, in isBackup > return self.getMetaParam(DMDK_CLASS) == BACKUP_DOMAIN > File "/usr/share/vdsm/storage/sd.py", line 722, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/persistentDict.py", line 89, in __getitem__ > return dec(self._dict[key]) > File "/usr/share/vdsm/storage/persistentDict.py", line 199, in __getitem__ > with self._accessWrapper(): > File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__ > return self.gen.next() > File "/usr/share/vdsm/storage/persistentDict.py", line 152, in _accessWrapper > self.refresh() > File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh > lines = self._metaRW.readlines() > File "/usr/share/vdsm/storage/fileSD.py", line 108, in readlines > return stripNewLines(self._oop.directReadLines(self._metafile)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-155::DEBUG::2014-04-11 22:02:37,396::task::595::TaskManager.Task::(_updateState) Task=`b4384bc6-414f-4e31-8119-6c98b49b3ec2`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:37,413::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-155::INFO::2014-04-11 22:02:37,456::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '56.2', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000268373', 'lastCheck': '4.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '125.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '17.6', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '125.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.4', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000257954', 'lastCheck': '3.5', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:02:37,469::task::1185::TaskManager.Task::(prepare) Task=`b4384bc6-414f-4e31-8119-6c98b49b3ec2`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '56.2', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000268373', 'lastCheck': '4.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '125.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '17.6', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '125.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.4', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000257954', 'lastCheck': '3.5', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:02:37,483::task::595::TaskManager.Task::(_updateState) Task=`b4384bc6-414f-4e31-8119-6c98b49b3ec2`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:37,496::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:37,509::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:37,525::task::990::TaskManager.Task::(_decref) Task=`b4384bc6-414f-4e31-8119-6c98b49b3ec2`::ref 0 aborting False >Thread-994::WARNING::2014-04-11 22:02:37,974::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-994::DEBUG::2014-04-11 22:02:38,001::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-996::WARNING::2014-04-11 22:02:37,999::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-995::WARNING::2014-04-11 22:02:38,005::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-996::DEBUG::2014-04-11 22:02:38,019::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-995::DEBUG::2014-04-11 22:02:38,025::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-31::ERROR::2014-04-11 22:02:38,036::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-31::ERROR::2014-04-11 22:02:38,045::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-1001::WARNING::2014-04-11 22:02:38,319::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-1001::DEBUG::2014-04-11 22:02:38,326::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-1003::WARNING::2014-04-11 22:02:38,342::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-1002::WARNING::2014-04-11 22:02:38,343::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-1003::DEBUG::2014-04-11 22:02:38,346::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-1002::DEBUG::2014-04-11 22:02:38,347::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-38::ERROR::2014-04-11 22:02:38,349::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-38::ERROR::2014-04-11 22:02:38,349::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-22::DEBUG::2014-04-11 22:02:42,879::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:02:42,891::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000293462 s, 1.3 MB/s\n'; <rc> = 0 >Thread-44::DEBUG::2014-04-11 22:02:43,989::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:02:43,999::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000218867 s, 3.4 MB/s\n'; <rc> = 0 >Thread-31::DEBUG::2014-04-11 22:02:48,056::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-31::DEBUG::2014-04-11 22:02:48,056::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-31::DEBUG::2014-04-11 22:02:48,057::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-31::DEBUG::2014-04-11 22:02:48,057::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-31::DEBUG::2014-04-11 22:02:48,058::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-31::DEBUG::2014-04-11 22:02:48,059::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-31::DEBUG::2014-04-11 22:02:48,113::misc::772::SamplingMethod::(__call__) Returning last result >Thread-31::DEBUG::2014-04-11 22:02:48,114::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-31::DEBUG::2014-04-11 22:02:48,255::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-31::DEBUG::2014-04-11 22:02:48,256::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-31::DEBUG::2014-04-11 22:02:48,257::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-31::DEBUG::2014-04-11 22:02:48,257::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-31::DEBUG::2014-04-11 22:02:48,258::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-31::DEBUG::2014-04-11 22:02:48,258::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-31::DEBUG::2014-04-11 22:02:48,259::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-31::DEBUG::2014-04-11 22:02:48,259::misc::772::SamplingMethod::(__call__) Returning last result >Thread-31::ERROR::2014-04-11 22:02:48,260::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-31::ERROR::2014-04-11 22:02:48,260::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-31::DEBUG::2014-04-11 22:02:48,261::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31::DEBUG::2014-04-11 22:02:48,264::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-38::ERROR::2014-04-11 22:02:48,350::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-38::ERROR::2014-04-11 22:02:48,351::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-38::DEBUG::2014-04-11 22:02:48,353::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-38::DEBUG::2014-04-11 22:02:48,355::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-31::DEBUG::2014-04-11 22:02:48,618::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-31::WARNING::2014-04-11 22:02:48,623::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-38::DEBUG::2014-04-11 22:02:48,700::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-38::WARNING::2014-04-11 22:02:48,704::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-38::DEBUG::2014-04-11 22:02:48,705::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::ERROR::2014-04-11 22:02:51,282::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-20::DEBUG::2014-04-11 22:02:51,282::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-20::INFO::2014-04-11 22:02:51,283::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-20::DEBUG::2014-04-11 22:02:51,284::domainMonitor::184::Storage.DomainMonitorThread::(_monitorLoop) Unable to release the host id 1 for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 181, in _monitorLoop > self.domain.releaseHostId(self.hostId, unused=True) > File "/usr/share/vdsm/storage/sd.py", line 461, in releaseHostId > self._clusterLock.releaseHostId(hostId, async, unused) > File "/usr/share/vdsm/storage/clusterlock.py", line 214, in releaseHostId > raise se.ReleaseHostIdFailure(self._sdUUID, e) >ReleaseHostIdFailure: Cannot release host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(115, 'Sanlock lockspace remove failure', 'Operation now in progress')) >MainThread::INFO::2014-04-11 22:02:51,285::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-22::DEBUG::2014-04-11 22:02:51,285::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >MainThread::INFO::2014-04-11 22:02:51,286::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-155::DEBUG::2014-04-11 22:02:53,262::task::595::TaskManager.Task::(_updateState) Task=`da557d9e-4d12-427a-b1f6-814981209639`::moving from state init -> state preparing >Thread-155::INFO::2014-04-11 22:02:53,262::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-155::INFO::2014-04-11 22:02:53,263::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '15.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.4', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '50.3', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000218867', 'lastCheck': '9.3', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:02:53,263::task::1185::TaskManager.Task::(prepare) Task=`da557d9e-4d12-427a-b1f6-814981209639`::finished: {'3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '15.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.4', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '50.3', 'valid': False}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000218867', 'lastCheck': '9.3', 'valid': True}} >Thread-155::DEBUG::2014-04-11 22:02:53,264::task::595::TaskManager.Task::(_updateState) Task=`da557d9e-4d12-427a-b1f6-814981209639`::moving from state preparing -> state finished >Thread-155::DEBUG::2014-04-11 22:02:53,264::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-155::DEBUG::2014-04-11 22:02:53,265::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-155::DEBUG::2014-04-11 22:02:53,265::task::990::TaskManager.Task::(_decref) Task=`da557d9e-4d12-427a-b1f6-814981209639`::ref 0 aborting False >Thread-44::DEBUG::2014-04-11 22:02:54,005::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 21619c8e-99ea-4813-be02-d708971e5393 >Thread-44::DEBUG::2014-04-11 22:02:54,082::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393 >Thread-44::DEBUG::2014-04-11 22:02:54,085::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-44::DEBUG::2014-04-11 22:02:54,101::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2'] >Thread-44::DEBUG::2014-04-11 22:02:54,104::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-44::INFO::2014-04-11 22:02:54,105::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 21619c8e-99ea-4813-be02-d708971e5393_imageNS already registered >Thread-44::INFO::2014-04-11 22:02:54,105::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 21619c8e-99ea-4813-be02-d708971e5393_volumeNS already registered >Thread-44::DEBUG::2014-04-11 22:02:54,138::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-44::DEBUG::2014-04-11 22:02:54,151::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000261909 s, 2.9 MB/s\n'; <rc> = 0 >MainThread::INFO::2014-04-11 22:03:00,061::vdsm::119::vds::(run) (PID: 33265) I am the actual vdsm 4.14.6-12.git49cacf0.el6 ashtivh01.suprtekstic.com (2.6.32-431.11.2.el6.x86_64) >MainThread::DEBUG::2014-04-11 22:03:00,062::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2014-04-11 22:03:00,063::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 >MainThread::WARNING::2014-04-11 22:03:00,070::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists >MainThread::DEBUG::2014-04-11 22:03:01,623::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >MainThread::DEBUG::2014-04-11 22:03:01,650::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving >MainThread::DEBUG::2014-04-11 22:03:01,650::hsm::424::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2014-04-11 22:03:01,931::hsm::424::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-04-11 22:03:01,931::hsm::448::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >MainThread::DEBUG::2014-04-11 22:03:01,941::hsm::480::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >MainThread::DEBUG::2014-04-11 22:03:01,941::hsm::481::Storage.HSM::(__cleanStorageRepository) Mount list: ['/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage', '/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage', '/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage', '/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso', '/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage'] >MainThread::DEBUG::2014-04-11 22:03:01,941::hsm::483::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >MainThread::DEBUG::2014-04-11 22:03:01,946::hsm::526::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >storageRefresh::DEBUG::2014-04-11 22:03:01,947::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >MainThread::INFO::2014-04-11 22:03:01,949::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >storageRefresh::DEBUG::2014-04-11 22:03:01,950::misc::764::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-04-11 22:03:02,005::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >storageRefresh::DEBUG::2014-04-11 22:03:02,008::misc::764::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-04-11 22:03:02,009::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >storageRefresh::DEBUG::2014-04-11 22:03:02,012::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:03:02,032::misc::772::SamplingMethod::(__call__) Returning last result >storageRefresh::DEBUG::2014-04-11 22:03:02,032::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >MainThread::DEBUG::2014-04-11 22:03:02,053::task::595::TaskManager.Task::(_updateState) Task=`20ba9a8b-8c3d-4503-9f6a-c7cf0e5b4607`::moving from state init -> state preparing >MainThread::INFO::2014-04-11 22:03:02,054::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x7f4eb80799e0>>) >MainThread::INFO::2014-04-11 22:03:02,054::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None >MainThread::DEBUG::2014-04-11 22:03:02,055::task::1185::TaskManager.Task::(prepare) Task=`20ba9a8b-8c3d-4503-9f6a-c7cf0e5b4607`::finished: None >MainThread::DEBUG::2014-04-11 22:03:02,055::task::595::TaskManager.Task::(_updateState) Task=`20ba9a8b-8c3d-4503-9f6a-c7cf0e5b4607`::moving from state preparing -> state finished >MainThread::DEBUG::2014-04-11 22:03:02,055::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2014-04-11 22:03:02,056::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2014-04-11 22:03:02,056::task::990::TaskManager.Task::(_decref) Task=`20ba9a8b-8c3d-4503-9f6a-c7cf0e5b4607`::ref 0 aborting False >MainThread::INFO::2014-04-11 22:03:02,071::momIF::47::MOM::(__init__) Starting up MOM >MainThread::INFO::2014-04-11 22:03:02,079::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds. >VM Channels Listener::INFO::2014-04-11 22:03:02,102::vmChannels::170::vds::(run) Starting VM channels listener thread. >clientIFinit::DEBUG::2014-04-11 22:03:02,103::libvirtconnection::145::root::(get) trying to connect libvirt >MainThread::WARNING::2014-04-11 22:03:02,110::clientIF::181::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed. >storageRefresh::DEBUG::2014-04-11 22:03:02,162::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:03:02,163::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,174::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,174::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,175::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,175::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,175::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,176::misc::772::SamplingMethod::(__call__) Returning last result >storageRefresh::DEBUG::2014-04-11 22:03:02,176::lvm::324::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,179::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:03:02,509::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:03:02,513::lvm::349::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,517::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,526::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:03:02,785::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:03:02,786::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:02,787::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:03:03,036::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:03:03,049::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:03:03,050::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:03:03,299::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:03:03,299::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::WARNING::2014-04-11 22:03:03,300::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists >storageRefresh::DEBUG::2014-04-11 22:03:03,300::hsm::392::Storage.HSM::(storageRefresh) HSM is ready >Thread-13::DEBUG::2014-04-11 22:03:05,731::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getCapabilities with () {} flowID [13a209e1] >Thread-13::DEBUG::2014-04-11 22:03:05,841::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-13::DEBUG::2014-04-11 22:03:05,857::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:03:05,905::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.6' (cwd None) >Thread-13::DEBUG::2014-04-11 22:03:05,915::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:03:05,921::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None) >Thread-13::DEBUG::2014-04-11 22:03:05,930::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:03:05,937::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.5' (cwd None) >Thread-13::DEBUG::2014-04-11 22:03:05,945::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:03:05,950::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None) >Thread-13::DEBUG::2014-04-11 22:03:05,958::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-13::DEBUG::2014-04-11 22:03:06,001::caps::458::root::(_getKeyPackages) rpm package glusterfs-rdma not found >Thread-13::DEBUG::2014-04-11 22:03:06,001::caps::458::root::(_getKeyPackages) rpm package glusterfs-fuse not found >Thread-13::DEBUG::2014-04-11 22:03:06,003::caps::458::root::(_getKeyPackages) rpm package gluster-swift not found >Thread-13::DEBUG::2014-04-11 22:03:06,011::caps::458::root::(_getKeyPackages) rpm package gluster-swift-object not found >Thread-13::DEBUG::2014-04-11 22:03:06,012::caps::458::root::(_getKeyPackages) rpm package glusterfs not found >Thread-13::DEBUG::2014-04-11 22:03:06,012::caps::458::root::(_getKeyPackages) rpm package gluster-swift-plugin not found >Thread-13::DEBUG::2014-04-11 22:03:06,022::caps::458::root::(_getKeyPackages) rpm package gluster-swift-account not found >Thread-13::DEBUG::2014-04-11 22:03:06,023::caps::458::root::(_getKeyPackages) rpm package gluster-swift-proxy not found >Thread-13::DEBUG::2014-04-11 22:03:06,023::caps::458::root::(_getKeyPackages) rpm package gluster-swift-doc not found >Thread-13::DEBUG::2014-04-11 22:03:06,024::caps::458::root::(_getKeyPackages) rpm package glusterfs-server not found >Thread-13::DEBUG::2014-04-11 22:03:06,024::caps::458::root::(_getKeyPackages) rpm package gluster-swift-container not found >Thread-13::DEBUG::2014-04-11 22:03:06,025::caps::458::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found >Thread-13::DEBUG::2014-04-11 22:03:06,050::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:93188a588127'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395791995.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '12.git49cacf0.el6', 'buildtime': 1397150575L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz', 'hooks': {'after_vm_start': {'50_promisc': {'md5': '5ea469fa06dd7cd1e52508672a7519db'}}, 'before_vm_migrate_source': {'50_sriov': {'md5': '47a59103b7c8b1439ae17dcd5ed197cd'}, '50_hostusb': {'md5': 'e50a0975acd8bafd0052b65a787c2fdd'}, '50_scratchpad': {'md5': 'be90995f959c60c5110c6a0bef67c7b8'}, '50_floppy': {'md5': 'c68fb91d44105c8b6f2d80566314027a'}}, 'after_device_destroy': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_vm_destroy': {'50_promisc': {'md5': 'dae0434ea8ca2492422a6fde5cc47a1e'}}, 'before_vm_start': {'50_sriov': {'md5': '3ebc60cd2e4eb089820102285fad7c45'}, '50_vmfex': {'md5': '9f5abb892ddb6b3daa779985d38d9f55'}, '50_qos': {'md5': 'cdf6f89d277a886691812ce1ec2fe105'}, '50_scratchpad': {'md5': '178171eb62ba95fc548e2450013f5ad5'}, '50_numa': {'md5': '5008c2826714ac5b63748780aabd2f25'}, '50_floppy': {'md5': '202fe18705a7d4c50c40c126e8f8dbe8'}, '50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}, '50_pincpu': {'md5': '0b5fb99ff0e7acb9ad534b87c02c59e3'}, '50_hugepages': {'md5': '6713d703f9d316f1664145b700cd7904'}, '50_fileinject': {'md5': '290dafd45601e010e0f496fa5b889a76'}, '50_macspoof': {'md5': '25deea559772719b90117f2e1009d823'}, '60_checkimages': {'md5': 'abd32c723a90ab2816e2163f5e02faf7'}, '10_faqemu': {'md5': 'e3cb338be6b9075a04bed0e7400ab7e2'}, '50_directlun': {'md5': 'dba5d80fdb7119ee0f4a40c26eac8448'}, '50_smbios': {'md5': '1a6bd8eddc3a86943e558bc4d3e73bd6'}, '50_vmdisk': {'md5': '67617c6e2d796388ddc6c065a97f714e'}, '50_hostusb': {'md5': '21c58568f3eae4f0d091005e0eff891a'}, '50_isolatedprivatevlan': {'md5': '0a25d7cd0b807ea5e0dfc3318ee98c91'}}, 'after_device_migrate_destination': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'after_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_vm_migrate_destination': {'50_vmfex': {'md5': 'eac2633f47fb309b4a5b7fe76531baec'}, '50_hugepages': {'md5': '6ddc484746297308b9f9acc5fcabaf10'}, '50_directlun': {'md5': '8a29ce41de079630cb58d48e280d55c2'}}, 'after_nic_hotunplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_vm_destroy': {'50_sriov': {'md5': '5890b2899d450495a62f3611fad3c526'}, '50_directlun': {'md5': '6d05c19e68a141eeb7ee5981d68c0cfa'}, '50_hostusb': {'md5': '0a37bf0a26546e3ab5e424b13653ce2b'}, '50_hugepages': {'md5': '4a980b78f8ee908bf2c61ea9bf156f4a'}, '50_scratchpad': {'md5': '316d21a669f97e83b485b45ab0b23eee'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0', 'addr': '172.16.10.1', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'bridged': False, 'interface': u'bond0', 'ipv6gateway': '::', 'gateway': '172.16.0.1'}, 'Blue': {'iface': 'Blue', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.6']}, 'Infrastructure': {'iface': 'Infrastructure', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.4']}, 'migration': {'iface': u'bond0.5', 'addr': '', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'bridged': False, 'interface': u'bond0.5', 'ipv6gateway': '::', 'gateway': ''}, 'Red': {'iface': 'Red', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.3']}}, 'bridges': {'Blue': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.6']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'Infrastructure': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.4']}, 'Red': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.3']}}, 'uuid': '4C4C4544-0059-5310-8031-B9C04F445A31', 'lastClientIface': 'bond0', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:5F', 'MASTER': 'bond0', 'DEVICE': 'em1', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:5F', 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:60', 'MASTER': 'bond0', 'DEVICE': 'em2', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:60', 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '12', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,lahf_lm,ida,arat,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:93188a588127', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '255.255.0.0', 'addr': '172.16.10.1', 'slaves': ['em1', 'em2'], 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.10.1', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond0', 'MTU': '1500', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32057', 'cpuSpeed': '1900.023', 'version_name': 'Snow Man', 'vlans': {'bond0.3': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Red', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 3, 'mtu': '1500'}, 'bond0.4': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Infrastructure', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 4, 'mtu': '1500'}, 'bond0.5': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'DEVICE': 'bond0.5', 'NM_CONTROLLED': 'no', 'VLAN': 'yes', 'DEFROUTE': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 5, 'mtu': '1500'}, 'bond0.6': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Blue', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.6', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 6, 'mtu': '1500'}}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '24', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.10.1'}} >Thread-13::DEBUG::2014-04-11 22:03:06,090::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getHardwareInfo with () {} flowID [13a209e1] >Thread-13::DEBUG::2014-04-11 22:03:06,097::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R420', 'systemSerialNumber': '9YS1DZ1', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0059-5310-8031-B9C04F445A31', 'systemManufacturer': 'Dell Inc.'}} >Thread-13::DEBUG::2014-04-11 22:03:06,278::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [5757a999] >Thread-13::DEBUG::2014-04-11 22:03:06,279::task::595::TaskManager.Task::(_updateState) Task=`b967b538-bfc8-4542-b173-e0038dad875f`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:03:06,280::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', conList=[{'connection': 'ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c', 'port': ''}, {'connection': 'ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/exports', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54', 'port': ''}, {'connection': 'ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'deb06d0b-14c6-490e-8fbb-552861511109', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/infrastructure', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac', 'port': ''}], options=None) >Thread-13::DEBUG::2014-04-11 22:03:06,288::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage >Thread-14::DEBUG::2014-04-11 22:03:09,742::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [61b945bb] >Thread-14::DEBUG::2014-04-11 22:03:09,745::task::595::TaskManager.Task::(_updateState) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:03:09,745::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:03:09,745::task::866::TaskManager.Task::(_setError) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:03:09,749::task::885::TaskManager.Task::(_run) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::Task._run: 483c539f-ff25-4895-ad9c-b51fdd90eb02 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:03:09,750::task::1211::TaskManager.Task::(stop) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:03:09,750::task::990::TaskManager.Task::(_decref) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:03:09,750::task::1168::TaskManager.Task::(prepare) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:03:09,751::task::1173::TaskManager.Task::(prepare) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:03:09,751::task::990::TaskManager.Task::(_decref) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:03:09,751::task::925::TaskManager.Task::(_doAbort) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:03:09,751::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:03:09,752::task::595::TaskManager.Task::(_updateState) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:03:09,752::task::550::TaskManager.Task::(__state_aborting) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:03:09,752::task::595::TaskManager.Task::(_updateState) Task=`483c539f-ff25-4895-ad9c-b51fdd90eb02`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:03:09,753::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:03:09,753::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:03:09,753::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:03:09,798::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [61b945bb] >Thread-14::DEBUG::2014-04-11 22:03:09,798::task::595::TaskManager.Task::(_updateState) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:03:09,799::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-14::ERROR::2014-04-11 22:03:09,799::task::866::TaskManager.Task::(_setError) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses > raise se.SpmStatusError() >SpmStatusError: Not SPM: () >Thread-14::DEBUG::2014-04-11 22:03:09,799::task::885::TaskManager.Task::(_run) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::Task._run: 6ea71689-0a3f-49da-8d73-f0a04c2a9144 () {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:03:09,800::task::1211::TaskManager.Task::(stop) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:03:09,800::task::990::TaskManager.Task::(_decref) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:03:09,800::task::1168::TaskManager.Task::(prepare) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::aborting: Task is aborted: 'Not SPM' - code 654 >Thread-14::DEBUG::2014-04-11 22:03:09,801::task::1173::TaskManager.Task::(prepare) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::Prepare: aborted: Not SPM >Thread-14::DEBUG::2014-04-11 22:03:09,801::task::990::TaskManager.Task::(_decref) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:03:09,801::task::925::TaskManager.Task::(_doAbort) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:03:09,801::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:03:09,802::task::595::TaskManager.Task::(_updateState) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:03:09,802::task::550::TaskManager.Task::(__state_aborting) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:03:09,802::task::595::TaskManager.Task::(_updateState) Task=`6ea71689-0a3f-49da-8d73-f0a04c2a9144`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:03:09,802::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:03:09,803::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:03:09,803::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}} >Thread-13::DEBUG::2014-04-11 22:04:06,298::hsm::2387::Storage.HSM::(connectStorageServer) prefetch failed: {} >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2384, in connectStorageServer > doms = self.__prefetchDomains(domType, conObj) > File "/usr/share/vdsm/storage/hsm.py", line 2321, in __prefetchDomains > goop.glob.glob(os.path.join(lPath, uuidPatern))) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-13::DEBUG::2014-04-11 22:04:06,299::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-13::DEBUG::2014-04-11 22:04:06,305::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage >Thread-13::DEBUG::2014-04-11 22:05:06,315::hsm::2387::Storage.HSM::(connectStorageServer) prefetch failed: {} >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2384, in connectStorageServer > doms = self.__prefetchDomains(domType, conObj) > File "/usr/share/vdsm/storage/hsm.py", line 2321, in __prefetchDomains > goop.glob.glob(os.path.join(lPath, uuidPatern))) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-13::DEBUG::2014-04-11 22:05:06,316::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-13::DEBUG::2014-04-11 22:05:06,320::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/exports /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_exports' (cwd None) >Thread-13::ERROR::2014-04-11 22:05:58,706::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect > self._mount.mount(self.options, self._vfsType) > File "/usr/share/vdsm/storage/mount.py", line 222, in mount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-13::ERROR::2014-04-11 22:05:58,720::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer > conObj.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect > return self._mountCon.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect > raise e >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-13::DEBUG::2014-04-11 22:05:58,720::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-13::DEBUG::2014-04-11 22:05:58,728::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage >Thread-14::DEBUG::2014-04-11 22:06:06,349::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:06:06,350::task::595::TaskManager.Task::(_updateState) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:06:06,350::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None) >Thread-14::DEBUG::2014-04-11 22:06:06,351::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`339a9a3f-9611-4ba8-be51-36469ce057f9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:06:06,351::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:06:06,352::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:06:06,352::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`339a9a3f-9611-4ba8-be51-36469ce057f9`::Granted request >Thread-14::DEBUG::2014-04-11 22:06:06,353::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`409e9708-1f83-4ae3-a368-59235998ce7f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:06:06,354::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:06:06,354::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:06:06,354::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`409e9708-1f83-4ae3-a368-59235998ce7f`::Granted request >Thread-14::INFO::2014-04-11 22:06:06,355::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c0b7950 >Thread-14::INFO::2014-04-11 22:06:06,355::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36) >Thread-14::DEBUG::2014-04-11 22:06:06,356::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,356::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,357::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,357::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,357::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,358::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,358::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:06:06,358::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:06:06,358::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 22:06:06,359::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:06:06,359::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 22:06:06,360::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 22:06:06,392::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:06:06,393::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:06:06,549::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:06:06,566::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,575::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,575::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,576::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,576::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,577::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,577::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::ERROR::2014-04-11 22:06:06,578::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::ERROR::2014-04-11 22:06:06,578::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:06:06,578::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:06:06,581::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-14::DEBUG::2014-04-11 22:06:06,928::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:06:06,930::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-14::DEBUG::2014-04-11 22:06:06,930::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-13::DEBUG::2014-04-11 22:06:58,738::hsm::2387::Storage.HSM::(connectStorageServer) prefetch failed: {} >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2384, in connectStorageServer > doms = self.__prefetchDomains(domType, conObj) > File "/usr/share/vdsm/storage/hsm.py", line 2321, in __prefetchDomains > goop.glob.glob(os.path.join(lPath, uuidPatern))) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-13::DEBUG::2014-04-11 22:06:58,739::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-13::DEBUG::2014-04-11 22:06:58,755::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/infrastructure /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure' (cwd None) >Thread-15::WARNING::2014-04-11 22:07:06,985::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-15::DEBUG::2014-04-11 22:07:07,004::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-16::WARNING::2014-04-11 22:07:07,002::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-16::DEBUG::2014-04-11 22:07:07,011::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-17::WARNING::2014-04-11 22:07:07,016::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-14::ERROR::2014-04-11 22:07:07,018::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',) >Thread-14::DEBUG::2014-04-11 22:07:07,019::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:07:07,020::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:07:07,020::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:07:07,021::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 22:07:07,021::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 22:07:07,021::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::DEBUG::2014-04-11 22:07:07,022::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:07:07,022::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-14::ERROR::2014-04-11 22:07:07,022::task::866::TaskManager.Task::(_setError) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool > spUUID, hostID, msdUUID, masterVersion, domainsMap) > File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool > res = pool.connect(hostID, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 625, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild > self.setMasterDomain(msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a' >Thread-14::DEBUG::2014-04-11 22:07:07,024::task::885::TaskManager.Task::(_run) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::Task._run: 2d141f7a-5323-436b-a21a-7bf94b55ce7d ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:07:07,024::task::1211::TaskManager.Task::(stop) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:07:07,025::task::990::TaskManager.Task::(_decref) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:07:07,025::task::1168::TaskManager.Task::(prepare) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-14::DEBUG::2014-04-11 22:07:07,025::task::1173::TaskManager.Task::(prepare) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::Prepare: aborted: Cannot find master domain >Thread-14::DEBUG::2014-04-11 22:07:07,026::task::990::TaskManager.Task::(_decref) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:07:07,026::task::925::TaskManager.Task::(_doAbort) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:07:07,026::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:07:07,027::task::595::TaskManager.Task::(_updateState) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:07:07,027::task::550::TaskManager.Task::(__state_aborting) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:07:07,027::task::595::TaskManager.Task::(_updateState) Task=`2d141f7a-5323-436b-a21a-7bf94b55ce7d`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:07:07,027::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:07:07,028::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:07:07,028::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}} >Thread-14::DEBUG::2014-04-11 22:07:07,054::task::595::TaskManager.Task::(_updateState) Task=`830a8c6f-5f48-44ee-9f03-0d789daef4a8`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:07:07,055::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:07:07,055::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-14::DEBUG::2014-04-11 22:07:07,055::task::1185::TaskManager.Task::(prepare) Task=`830a8c6f-5f48-44ee-9f03-0d789daef4a8`::finished: {} >Thread-14::DEBUG::2014-04-11 22:07:07,055::task::595::TaskManager.Task::(_updateState) Task=`830a8c6f-5f48-44ee-9f03-0d789daef4a8`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:07:07,056::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:07:07,056::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:07:07,056::task::990::TaskManager.Task::(_decref) Task=`830a8c6f-5f48-44ee-9f03-0d789daef4a8`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:07:07,793::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call setMOMPolicyParameters with ({'balloonEnabled': True, 'ksmEnabled': True},) {} flowID [5757a999] >Thread-14::DEBUG::2014-04-11 22:07:07,889::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}} >Thread-14::DEBUG::2014-04-11 22:07:10,106::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4bd6819a] >Thread-14::DEBUG::2014-04-11 22:07:10,124::task::595::TaskManager.Task::(_updateState) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:07:10,124::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:07:10,124::task::866::TaskManager.Task::(_setError) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:07:10,125::task::885::TaskManager.Task::(_run) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::Task._run: 82312493-60c4-4805-82f6-c4ddd861a537 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:07:10,125::task::1211::TaskManager.Task::(stop) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:07:10,125::task::990::TaskManager.Task::(_decref) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:07:10,126::task::1168::TaskManager.Task::(prepare) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:07:10,126::task::1173::TaskManager.Task::(prepare) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:07:10,126::task::990::TaskManager.Task::(_decref) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:07:10,127::task::925::TaskManager.Task::(_doAbort) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:07:10,127::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:07:10,127::task::595::TaskManager.Task::(_updateState) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:07:10,127::task::550::TaskManager.Task::(__state_aborting) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:07:10,128::task::595::TaskManager.Task::(_updateState) Task=`82312493-60c4-4805-82f6-c4ddd861a537`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:07:10,128::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:07:10,128::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:07:10,128::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:07:10,172::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4bd6819a] >Thread-14::DEBUG::2014-04-11 22:07:10,173::task::595::TaskManager.Task::(_updateState) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:07:10,173::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None) >Thread-14::DEBUG::2014-04-11 22:07:10,174::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7c1e1b75-f19a-4cae-b0ac-c75dca994ad1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:07:10,174::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:07:10,175::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:07:10,175::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`7c1e1b75-f19a-4cae-b0ac-c75dca994ad1`::Granted request >Thread-14::DEBUG::2014-04-11 22:07:10,176::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`1c59fff7-04d8-4b73-891a-7bbc2c459a94`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:07:10,176::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:07:10,176::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:07:10,177::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`1c59fff7-04d8-4b73-891a-7bbc2c459a94`::Granted request >Thread-14::INFO::2014-04-11 22:07:10,177::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e840aec20 >Thread-14::INFO::2014-04-11 22:07:10,178::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36) >Thread-14::DEBUG::2014-04-11 22:07:10,178::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,178::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,179::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,179::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,180::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,180::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,180::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:07:10,180::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:07:10,181::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 22:07:10,181::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:07:10,181::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 22:07:10,182::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 22:07:10,208::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:07:10,209::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:07:10,344::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:07:10,345::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,353::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,354::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,354::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,355::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,355::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,356::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::ERROR::2014-04-11 22:07:10,356::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::ERROR::2014-04-11 22:07:10,356::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:07:10,357::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:07:10,360::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-14::DEBUG::2014-04-11 22:07:10,717::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:07:10,719::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-14::DEBUG::2014-04-11 22:07:10,720::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:07:11,049::task::595::TaskManager.Task::(_updateState) Task=`debd47f8-c976-4371-b40e-59f43a870074`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:07:11,049::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:07:11,050::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:07:11,050::task::1185::TaskManager.Task::(prepare) Task=`debd47f8-c976-4371-b40e-59f43a870074`::finished: {} >Thread-28::DEBUG::2014-04-11 22:07:11,050::task::595::TaskManager.Task::(_updateState) Task=`debd47f8-c976-4371-b40e-59f43a870074`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:07:11,051::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:07:11,051::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:07:11,051::task::990::TaskManager.Task::(_decref) Task=`debd47f8-c976-4371-b40e-59f43a870074`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:07:26,386::task::595::TaskManager.Task::(_updateState) Task=`2a2785ff-ef1f-4b6f-9bb4-27854056fc36`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:07:26,398::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:07:26,399::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:07:26,399::task::1185::TaskManager.Task::(prepare) Task=`2a2785ff-ef1f-4b6f-9bb4-27854056fc36`::finished: {} >Thread-28::DEBUG::2014-04-11 22:07:26,399::task::595::TaskManager.Task::(_updateState) Task=`2a2785ff-ef1f-4b6f-9bb4-27854056fc36`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:07:26,400::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:07:26,400::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:07:26,400::task::990::TaskManager.Task::(_decref) Task=`2a2785ff-ef1f-4b6f-9bb4-27854056fc36`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:07:41,707::task::595::TaskManager.Task::(_updateState) Task=`bc860597-8031-4590-8e9f-01687949ad49`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:07:41,707::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:07:41,708::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:07:41,708::task::1185::TaskManager.Task::(prepare) Task=`bc860597-8031-4590-8e9f-01687949ad49`::finished: {} >Thread-28::DEBUG::2014-04-11 22:07:41,709::task::595::TaskManager.Task::(_updateState) Task=`bc860597-8031-4590-8e9f-01687949ad49`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:07:41,709::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:07:41,709::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:07:41,710::task::990::TaskManager.Task::(_decref) Task=`bc860597-8031-4590-8e9f-01687949ad49`::ref 0 aborting False >Thread-13::ERROR::2014-04-11 22:07:51,049::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect > self._mount.mount(self.options, self._vfsType) > File "/usr/share/vdsm/storage/mount.py", line 222, in mount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-13::ERROR::2014-04-11 22:07:51,062::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer > conObj.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect > return self._mountCon.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect > raise e >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-13::DEBUG::2014-04-11 22:07:51,062::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {} >Thread-13::DEBUG::2014-04-11 22:07:51,074::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso >Thread-13::DEBUG::2014-04-11 22:07:51,372::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',) >Thread-13::DEBUG::2014-04-11 22:07:51,373::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain} >Thread-13::DEBUG::2014-04-11 22:07:51,382::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage >Thread-13::DEBUG::2014-04-11 22:07:51,385::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('21619c8e-99ea-4813-be02-d708971e5393',) >Thread-13::DEBUG::2014-04-11 22:07:51,385::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain} >Thread-13::INFO::2014-04-11 22:07:51,386::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]} >Thread-13::DEBUG::2014-04-11 22:07:51,386::task::1185::TaskManager.Task::(prepare) Task=`b967b538-bfc8-4542-b173-e0038dad875f`::finished: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]} >Thread-13::DEBUG::2014-04-11 22:07:51,386::task::595::TaskManager.Task::(_updateState) Task=`b967b538-bfc8-4542-b173-e0038dad875f`::moving from state preparing -> state finished >Thread-13::DEBUG::2014-04-11 22:07:51,387::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:07:51,387::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:07:51,388::task::990::TaskManager.Task::(_decref) Task=`b967b538-bfc8-4542-b173-e0038dad875f`::ref 0 aborting False >Thread-13::ERROR::2014-04-11 22:07:51,389::SecureXMLRPCServer::168::root::(handle_error) client ('172.16.10.1', 59605) >Traceback (most recent call last): > File "/usr/lib64/python2.6/SocketServer.py", line 570, in process_request_thread > self.finish_request(request, client_address) > File "/usr/lib64/python2.6/SocketServer.py", line 332, in finish_request > self.RequestHandlerClass(request, client_address, self) > File "/usr/lib64/python2.6/SocketServer.py", line 627, in __init__ > self.handle() > File "/usr/lib64/python2.6/BaseHTTPServer.py", line 331, in handle > self.handle_one_request() > File "/usr/lib64/python2.6/BaseHTTPServer.py", line 323, in handle_one_request > method() > File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 219, in do_POST > self.send_response(200) > File "/usr/lib64/python2.6/BaseHTTPServer.py", line 384, in send_response > self.send_header('Server', self.version_string()) > File "/usr/lib64/python2.6/BaseHTTPServer.py", line 390, in send_header > self.wfile.write("%s: %s\r\n" % (keyword, value)) > File "/usr/lib64/python2.6/socket.py", line 324, in write > self.flush() > File "/usr/lib64/python2.6/socket.py", line 303, in flush > self._sock.sendall(buffer(data, write_offset, buffer_size)) > File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 222, in write > return self._write_bio(data) > File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 205, in _write_bio > return m2.ssl_write(self.ssl, data, self._timeout) >SSLError: (32, 'Broken pipe') >Thread-28::DEBUG::2014-04-11 22:07:57,026::task::595::TaskManager.Task::(_updateState) Task=`4e7aedfa-64a4-48ab-8c05-e5c5ef19b2c7`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:07:57,027::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:07:57,027::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:07:57,027::task::1185::TaskManager.Task::(prepare) Task=`4e7aedfa-64a4-48ab-8c05-e5c5ef19b2c7`::finished: {} >Thread-28::DEBUG::2014-04-11 22:07:57,028::task::595::TaskManager.Task::(_updateState) Task=`4e7aedfa-64a4-48ab-8c05-e5c5ef19b2c7`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:07:57,028::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:07:57,029::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:07:57,029::task::990::TaskManager.Task::(_decref) Task=`4e7aedfa-64a4-48ab-8c05-e5c5ef19b2c7`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:07:59,493::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [70476e89] >Thread-28::DEBUG::2014-04-11 22:07:59,493::task::595::TaskManager.Task::(_updateState) Task=`df78d592-6a16-4082-9277-2a886613ef82`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:07:59,494::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c', 'port': ''}], options=None) >Thread-28::DEBUG::2014-04-11 22:07:59,501::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage >Thread-21::WARNING::2014-04-11 22:08:10,773::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-21::DEBUG::2014-04-11 22:08:10,779::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-22::WARNING::2014-04-11 22:08:10,779::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-23::WARNING::2014-04-11 22:08:10,783::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-22::DEBUG::2014-04-11 22:08:10,787::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-23::DEBUG::2014-04-11 22:08:10,788::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-14::ERROR::2014-04-11 22:08:10,791::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',) >Thread-14::DEBUG::2014-04-11 22:08:10,793::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:08:10,793::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:08:10,793::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:08:10,794::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 22:08:10,794::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 22:08:10,795::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::DEBUG::2014-04-11 22:08:10,795::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:08:10,795::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-14::ERROR::2014-04-11 22:08:10,796::task::866::TaskManager.Task::(_setError) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool > spUUID, hostID, msdUUID, masterVersion, domainsMap) > File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool > res = pool.connect(hostID, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 625, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild > self.setMasterDomain(msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a' >Thread-14::DEBUG::2014-04-11 22:08:10,797::task::885::TaskManager.Task::(_run) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::Task._run: 340cfcbe-3af8-4eb8-ab58-0f46c1e7601e ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:08:10,797::task::1211::TaskManager.Task::(stop) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:08:10,798::task::990::TaskManager.Task::(_decref) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:08:10,798::task::1168::TaskManager.Task::(prepare) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-14::DEBUG::2014-04-11 22:08:10,798::task::1173::TaskManager.Task::(prepare) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::Prepare: aborted: Cannot find master domain >Thread-14::DEBUG::2014-04-11 22:08:10,799::task::990::TaskManager.Task::(_decref) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:08:10,799::task::925::TaskManager.Task::(_doAbort) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:08:10,799::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:08:10,800::task::595::TaskManager.Task::(_updateState) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:08:10,800::task::550::TaskManager.Task::(__state_aborting) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:08:10,800::task::595::TaskManager.Task::(_updateState) Task=`340cfcbe-3af8-4eb8-ab58-0f46c1e7601e`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:08:10,801::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:08:10,801::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:08:10,801::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}} >Thread-14::DEBUG::2014-04-11 22:08:20,904::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6ead44fe] >Thread-14::DEBUG::2014-04-11 22:08:20,907::task::595::TaskManager.Task::(_updateState) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:08:20,907::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:08:20,908::task::866::TaskManager.Task::(_setError) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:08:20,909::task::885::TaskManager.Task::(_run) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::Task._run: 070a06b5-72bb-4ce6-8480-7a56e606b17d ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:08:20,909::task::1211::TaskManager.Task::(stop) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:08:20,910::task::990::TaskManager.Task::(_decref) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:08:20,910::task::1168::TaskManager.Task::(prepare) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:08:20,910::task::1173::TaskManager.Task::(prepare) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:08:20,911::task::990::TaskManager.Task::(_decref) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:08:20,911::task::925::TaskManager.Task::(_doAbort) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:08:20,911::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:08:20,911::task::595::TaskManager.Task::(_updateState) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:08:20,912::task::550::TaskManager.Task::(__state_aborting) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:08:20,912::task::595::TaskManager.Task::(_updateState) Task=`070a06b5-72bb-4ce6-8480-7a56e606b17d`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:08:20,912::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:08:20,913::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:08:20,913::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:08:20,917::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6ead44fe] >Thread-14::DEBUG::2014-04-11 22:08:20,917::task::595::TaskManager.Task::(_updateState) Task=`15771a66-3978-4458-88d7-ac47a2424756`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:08:20,918::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-14::ERROR::2014-04-11 22:08:20,918::task::866::TaskManager.Task::(_setError) Task=`15771a66-3978-4458-88d7-ac47a2424756`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses > raise se.SpmStatusError() >SpmStatusError: Not SPM: () >Thread-14::DEBUG::2014-04-11 22:08:20,919::task::885::TaskManager.Task::(_run) Task=`15771a66-3978-4458-88d7-ac47a2424756`::Task._run: 15771a66-3978-4458-88d7-ac47a2424756 () {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:08:20,919::task::1211::TaskManager.Task::(stop) Task=`15771a66-3978-4458-88d7-ac47a2424756`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:08:20,919::task::990::TaskManager.Task::(_decref) Task=`15771a66-3978-4458-88d7-ac47a2424756`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:08:20,919::task::1168::TaskManager.Task::(prepare) Task=`15771a66-3978-4458-88d7-ac47a2424756`::aborting: Task is aborted: 'Not SPM' - code 654 >Thread-14::DEBUG::2014-04-11 22:08:20,920::task::1173::TaskManager.Task::(prepare) Task=`15771a66-3978-4458-88d7-ac47a2424756`::Prepare: aborted: Not SPM >Thread-14::DEBUG::2014-04-11 22:08:20,920::task::990::TaskManager.Task::(_decref) Task=`15771a66-3978-4458-88d7-ac47a2424756`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:08:20,920::task::925::TaskManager.Task::(_doAbort) Task=`15771a66-3978-4458-88d7-ac47a2424756`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:08:20,921::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:08:20,921::task::595::TaskManager.Task::(_updateState) Task=`15771a66-3978-4458-88d7-ac47a2424756`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:08:20,921::task::550::TaskManager.Task::(__state_aborting) Task=`15771a66-3978-4458-88d7-ac47a2424756`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:08:20,921::task::595::TaskManager.Task::(_updateState) Task=`15771a66-3978-4458-88d7-ac47a2424756`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:08:20,922::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:08:20,922::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:08:20,922::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}} >Thread-14::DEBUG::2014-04-11 22:08:21,038::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6ead44fe] >Thread-14::DEBUG::2014-04-11 22:08:21,039::task::595::TaskManager.Task::(_updateState) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:08:21,039::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:08:21,040::task::866::TaskManager.Task::(_setError) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:08:21,042::task::885::TaskManager.Task::(_run) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::Task._run: caf5b536-8710-4bfe-add0-55a59758cf5a ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:08:21,042::task::1211::TaskManager.Task::(stop) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:08:21,042::task::990::TaskManager.Task::(_decref) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:08:21,043::task::1168::TaskManager.Task::(prepare) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:08:21,043::task::1173::TaskManager.Task::(prepare) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:08:21,043::task::990::TaskManager.Task::(_decref) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:08:21,044::task::925::TaskManager.Task::(_doAbort) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:08:21,044::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:08:21,044::task::595::TaskManager.Task::(_updateState) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:08:21,044::task::550::TaskManager.Task::(__state_aborting) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:08:21,045::task::595::TaskManager.Task::(_updateState) Task=`caf5b536-8710-4bfe-add0-55a59758cf5a`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:08:21,045::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:08:21,045::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:08:21,046::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:08:21,090::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6ead44fe] >Thread-14::DEBUG::2014-04-11 22:08:21,091::task::595::TaskManager.Task::(_updateState) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:08:21,091::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None) >Thread-14::DEBUG::2014-04-11 22:08:21,094::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f1375f11-a4c2-4e61-92d7-9e812132ebb6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:08:21,094::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:08:21,095::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:08:21,095::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f1375f11-a4c2-4e61-92d7-9e812132ebb6`::Granted request >Thread-14::DEBUG::2014-04-11 22:08:21,096::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`3406dec5-f6c1-44f4-b2a3-d576719a6ddb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:08:21,096::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:08:21,097::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:08:21,097::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`3406dec5-f6c1-44f4-b2a3-d576719a6ddb`::Granted request >Thread-14::INFO::2014-04-11 22:08:21,098::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c0b7170 >Thread-14::INFO::2014-04-11 22:08:21,098::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36) >Thread-14::DEBUG::2014-04-11 22:08:21,098::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,099::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,099::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,099::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,100::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,100::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,101::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:08:21,101::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:08:21,101::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 22:08:21,101::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:08:21,102::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 22:08:21,102::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 22:08:21,163::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:08:21,164::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:08:21,286::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:08:21,287::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,287::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,288::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,288::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,289::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,289::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,289::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::ERROR::2014-04-11 22:08:21,290::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::ERROR::2014-04-11 22:08:21,290::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:08:21,290::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:08:21,294::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-14::DEBUG::2014-04-11 22:08:21,635::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:08:21,646::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-14::DEBUG::2014-04-11 22:08:21,647::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:08:59,502::hsm::2387::Storage.HSM::(connectStorageServer) prefetch failed: {'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': <function findDomain at 0x1b75230>, '21619c8e-99ea-4813-be02-d708971e5393': <function findDomain at 0x1b75230>} >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2384, in connectStorageServer > doms = self.__prefetchDomains(domType, conObj) > File "/usr/share/vdsm/storage/hsm.py", line 2321, in __prefetchDomains > goop.glob.glob(os.path.join(lPath, uuidPatern))) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-28::DEBUG::2014-04-11 22:08:59,503::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain} >Thread-28::INFO::2014-04-11 22:08:59,504::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}]} >Thread-28::DEBUG::2014-04-11 22:08:59,516::task::1185::TaskManager.Task::(prepare) Task=`df78d592-6a16-4082-9277-2a886613ef82`::finished: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}]} >Thread-28::DEBUG::2014-04-11 22:08:59,517::task::595::TaskManager.Task::(_updateState) Task=`df78d592-6a16-4082-9277-2a886613ef82`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:08:59,519::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:08:59,520::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:08:59,522::task::990::TaskManager.Task::(_decref) Task=`df78d592-6a16-4082-9277-2a886613ef82`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:08:59,548::task::595::TaskManager.Task::(_updateState) Task=`1eed29ad-0607-4037-9d60-20db8b7863d5`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:08:59,551::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:08:59,552::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:08:59,554::task::1185::TaskManager.Task::(prepare) Task=`1eed29ad-0607-4037-9d60-20db8b7863d5`::finished: {} >Thread-28::DEBUG::2014-04-11 22:08:59,556::task::595::TaskManager.Task::(_updateState) Task=`1eed29ad-0607-4037-9d60-20db8b7863d5`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:08:59,557::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:08:59,559::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:08:59,561::task::990::TaskManager.Task::(_decref) Task=`1eed29ad-0607-4037-9d60-20db8b7863d5`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:09:14,923::task::595::TaskManager.Task::(_updateState) Task=`584ed49f-bacf-42df-8d52-eb7c41eb68cb`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:09:14,924::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:09:14,924::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:09:14,925::task::1185::TaskManager.Task::(prepare) Task=`584ed49f-bacf-42df-8d52-eb7c41eb68cb`::finished: {} >Thread-28::DEBUG::2014-04-11 22:09:14,925::task::595::TaskManager.Task::(_updateState) Task=`584ed49f-bacf-42df-8d52-eb7c41eb68cb`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:09:14,925::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:09:14,926::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:09:14,926::task::990::TaskManager.Task::(_decref) Task=`584ed49f-bacf-42df-8d52-eb7c41eb68cb`::ref 0 aborting False >Thread-29::WARNING::2014-04-11 22:09:21,697::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-30::WARNING::2014-04-11 22:09:21,703::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-29::DEBUG::2014-04-11 22:09:21,708::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-31::WARNING::2014-04-11 22:09:21,706::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-30::DEBUG::2014-04-11 22:09:21,711::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-31::DEBUG::2014-04-11 22:09:21,715::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-14::ERROR::2014-04-11 22:09:21,717::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',) >Thread-14::DEBUG::2014-04-11 22:09:21,718::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:09:21,718::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:09:21,719::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:09:21,719::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 22:09:21,720::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 22:09:21,720::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::DEBUG::2014-04-11 22:09:21,720::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:09:21,721::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-14::ERROR::2014-04-11 22:09:21,721::task::866::TaskManager.Task::(_setError) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool > spUUID, hostID, msdUUID, masterVersion, domainsMap) > File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool > res = pool.connect(hostID, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 625, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild > self.setMasterDomain(msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a' >Thread-14::DEBUG::2014-04-11 22:09:21,722::task::885::TaskManager.Task::(_run) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::Task._run: 09e8e71b-c838-4573-a0ef-59f3063aa1fc ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:09:21,722::task::1211::TaskManager.Task::(stop) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:09:21,723::task::990::TaskManager.Task::(_decref) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:09:21,723::task::1168::TaskManager.Task::(prepare) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-14::DEBUG::2014-04-11 22:09:21,724::task::1173::TaskManager.Task::(prepare) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::Prepare: aborted: Cannot find master domain >Thread-14::DEBUG::2014-04-11 22:09:21,724::task::990::TaskManager.Task::(_decref) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:09:21,724::task::925::TaskManager.Task::(_doAbort) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:09:21,725::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:09:21,725::task::595::TaskManager.Task::(_updateState) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:09:21,725::task::550::TaskManager.Task::(__state_aborting) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:09:21,726::task::595::TaskManager.Task::(_updateState) Task=`09e8e71b-c838-4573-a0ef-59f3063aa1fc`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:09:21,726::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:09:21,726::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:09:21,727::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}} >Thread-14::DEBUG::2014-04-11 22:09:21,887::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [7de167af] >Thread-14::DEBUG::2014-04-11 22:09:21,888::task::595::TaskManager.Task::(_updateState) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:09:21,889::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:09:21,889::task::866::TaskManager.Task::(_setError) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:09:21,890::task::885::TaskManager.Task::(_run) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::Task._run: 8bcbb6aa-d71f-44cd-bdcb-47c07661154d ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:09:21,890::task::1211::TaskManager.Task::(stop) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:09:21,891::task::990::TaskManager.Task::(_decref) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:09:21,891::task::1168::TaskManager.Task::(prepare) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:09:21,891::task::1173::TaskManager.Task::(prepare) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:09:21,892::task::990::TaskManager.Task::(_decref) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:09:21,892::task::925::TaskManager.Task::(_doAbort) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:09:21,892::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:09:21,892::task::595::TaskManager.Task::(_updateState) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:09:21,893::task::550::TaskManager.Task::(__state_aborting) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:09:21,893::task::595::TaskManager.Task::(_updateState) Task=`8bcbb6aa-d71f-44cd-bdcb-47c07661154d`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:09:21,893::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:09:21,894::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:09:21,894::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:09:21,942::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [7de167af] >Thread-14::DEBUG::2014-04-11 22:09:21,942::task::595::TaskManager.Task::(_updateState) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:09:21,943::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None) >Thread-14::DEBUG::2014-04-11 22:09:21,944::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4e400ada-cd83-4e80-b309-ff62c3f5ae9c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:09:21,944::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:09:21,944::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:09:21,945::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4e400ada-cd83-4e80-b309-ff62c3f5ae9c`::Granted request >Thread-14::DEBUG::2014-04-11 22:09:21,946::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`e490f3fa-3f5b-49b7-ae55-87e7e63dd91e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:09:21,946::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:09:21,947::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:09:21,947::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`e490f3fa-3f5b-49b7-ae55-87e7e63dd91e`::Granted request >Thread-14::INFO::2014-04-11 22:09:21,948::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c12f908 >Thread-14::INFO::2014-04-11 22:09:21,948::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36) >Thread-14::DEBUG::2014-04-11 22:09:21,948::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:21,949::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:21,949::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:21,950::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:21,950::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:21,951::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:21,951::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:09:21,951::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:09:21,952::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 22:09:21,952::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:09:21,952::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 22:09:21,953::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 22:09:21,996::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:09:21,997::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:09:22,126::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:09:22,127::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:22,128::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:22,128::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:22,129::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:22,129::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:22,129::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:22,130::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::ERROR::2014-04-11 22:09:22,130::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::ERROR::2014-04-11 22:09:22,131::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:09:22,131::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:09:22,134::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-14::DEBUG::2014-04-11 22:09:22,498::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:09:22,500::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-14::DEBUG::2014-04-11 22:09:22,501::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:09:30,241::task::595::TaskManager.Task::(_updateState) Task=`6dcc0826-8170-4b76-a0de-7918118dd922`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:09:30,241::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:09:30,242::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:09:30,242::task::1185::TaskManager.Task::(prepare) Task=`6dcc0826-8170-4b76-a0de-7918118dd922`::finished: {} >Thread-28::DEBUG::2014-04-11 22:09:30,242::task::595::TaskManager.Task::(_updateState) Task=`6dcc0826-8170-4b76-a0de-7918118dd922`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:09:30,243::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:09:30,243::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:09:30,243::task::990::TaskManager.Task::(_decref) Task=`6dcc0826-8170-4b76-a0de-7918118dd922`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:09:45,574::task::595::TaskManager.Task::(_updateState) Task=`42d87c0c-63f1-4161-81a4-2d53a2008b7a`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:09:45,575::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:09:45,575::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:09:45,576::task::1185::TaskManager.Task::(prepare) Task=`42d87c0c-63f1-4161-81a4-2d53a2008b7a`::finished: {} >Thread-28::DEBUG::2014-04-11 22:09:45,576::task::595::TaskManager.Task::(_updateState) Task=`42d87c0c-63f1-4161-81a4-2d53a2008b7a`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:09:45,576::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:09:45,576::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:09:45,577::task::990::TaskManager.Task::(_decref) Task=`42d87c0c-63f1-4161-81a4-2d53a2008b7a`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:10:00,897::task::595::TaskManager.Task::(_updateState) Task=`aa06387d-ee84-4265-bade-34daf29bdd59`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:10:00,898::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:10:00,898::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:10:00,898::task::1185::TaskManager.Task::(prepare) Task=`aa06387d-ee84-4265-bade-34daf29bdd59`::finished: {} >Thread-28::DEBUG::2014-04-11 22:10:00,899::task::595::TaskManager.Task::(_updateState) Task=`aa06387d-ee84-4265-bade-34daf29bdd59`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:10:00,899::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:10:00,899::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:10:00,900::task::990::TaskManager.Task::(_decref) Task=`aa06387d-ee84-4265-bade-34daf29bdd59`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:10:16,313::task::595::TaskManager.Task::(_updateState) Task=`2c0b755b-70ea-4d54-9aba-4b36bfef7db8`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:10:16,314::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:10:16,314::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:10:16,315::task::1185::TaskManager.Task::(prepare) Task=`2c0b755b-70ea-4d54-9aba-4b36bfef7db8`::finished: {} >Thread-28::DEBUG::2014-04-11 22:10:16,315::task::595::TaskManager.Task::(_updateState) Task=`2c0b755b-70ea-4d54-9aba-4b36bfef7db8`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:10:16,315::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:10:16,316::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:10:16,316::task::990::TaskManager.Task::(_decref) Task=`2c0b755b-70ea-4d54-9aba-4b36bfef7db8`::ref 0 aborting False >Thread-35::WARNING::2014-04-11 22:10:22,567::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-36::WARNING::2014-04-11 22:10:22,573::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-35::DEBUG::2014-04-11 22:10:22,580::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-37::WARNING::2014-04-11 22:10:22,575::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-36::DEBUG::2014-04-11 22:10:22,583::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-37::DEBUG::2014-04-11 22:10:22,587::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-14::ERROR::2014-04-11 22:10:22,589::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',) >Thread-14::DEBUG::2014-04-11 22:10:22,590::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:10:22,591::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:10:22,591::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:10:22,591::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 22:10:22,592::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 22:10:22,592::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::DEBUG::2014-04-11 22:10:22,592::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:10:22,593::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-14::ERROR::2014-04-11 22:10:22,593::task::866::TaskManager.Task::(_setError) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool > spUUID, hostID, msdUUID, masterVersion, domainsMap) > File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool > res = pool.connect(hostID, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 625, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild > self.setMasterDomain(msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a' >Thread-14::DEBUG::2014-04-11 22:10:22,594::task::885::TaskManager.Task::(_run) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::Task._run: e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:10:22,594::task::1211::TaskManager.Task::(stop) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:10:22,595::task::990::TaskManager.Task::(_decref) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:10:22,595::task::1168::TaskManager.Task::(prepare) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-14::DEBUG::2014-04-11 22:10:22,596::task::1173::TaskManager.Task::(prepare) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::Prepare: aborted: Cannot find master domain >Thread-14::DEBUG::2014-04-11 22:10:22,596::task::990::TaskManager.Task::(_decref) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:10:22,596::task::925::TaskManager.Task::(_doAbort) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:10:22,597::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:10:22,597::task::595::TaskManager.Task::(_updateState) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:10:22,597::task::550::TaskManager.Task::(__state_aborting) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:10:22,597::task::595::TaskManager.Task::(_updateState) Task=`e5dbd81a-1c5e-4ec1-aeec-0c61917ec62d`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:10:22,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:10:22,598::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:10:22,598::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}} >Thread-14::DEBUG::2014-04-11 22:10:22,690::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4406499a] >Thread-14::DEBUG::2014-04-11 22:10:22,691::task::595::TaskManager.Task::(_updateState) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:10:22,692::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:10:22,694::task::866::TaskManager.Task::(_setError) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:10:22,695::task::885::TaskManager.Task::(_run) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::Task._run: aceb131b-cbe1-412a-b2e1-4ea192323454 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:10:22,696::task::1211::TaskManager.Task::(stop) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:10:22,696::task::990::TaskManager.Task::(_decref) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:10:22,696::task::1168::TaskManager.Task::(prepare) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:10:22,697::task::1173::TaskManager.Task::(prepare) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:10:22,697::task::990::TaskManager.Task::(_decref) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:10:22,697::task::925::TaskManager.Task::(_doAbort) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:10:22,698::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:10:22,698::task::595::TaskManager.Task::(_updateState) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:10:22,698::task::550::TaskManager.Task::(__state_aborting) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:10:22,699::task::595::TaskManager.Task::(_updateState) Task=`aceb131b-cbe1-412a-b2e1-4ea192323454`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:10:22,699::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:10:22,699::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:10:22,700::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:10:22,704::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4406499a] >Thread-14::DEBUG::2014-04-11 22:10:22,704::task::595::TaskManager.Task::(_updateState) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:10:22,705::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-14::ERROR::2014-04-11 22:10:22,705::task::866::TaskManager.Task::(_setError) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses > raise se.SpmStatusError() >SpmStatusError: Not SPM: () >Thread-14::DEBUG::2014-04-11 22:10:22,706::task::885::TaskManager.Task::(_run) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::Task._run: 689eb1db-cb02-49a5-8d58-bff01b09eb91 () {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:10:22,706::task::1211::TaskManager.Task::(stop) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:10:22,707::task::990::TaskManager.Task::(_decref) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:10:22,707::task::1168::TaskManager.Task::(prepare) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::aborting: Task is aborted: 'Not SPM' - code 654 >Thread-14::DEBUG::2014-04-11 22:10:22,707::task::1173::TaskManager.Task::(prepare) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::Prepare: aborted: Not SPM >Thread-14::DEBUG::2014-04-11 22:10:22,707::task::990::TaskManager.Task::(_decref) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:10:22,708::task::925::TaskManager.Task::(_doAbort) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:10:22,708::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:10:22,708::task::595::TaskManager.Task::(_updateState) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:10:22,709::task::550::TaskManager.Task::(__state_aborting) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:10:22,709::task::595::TaskManager.Task::(_updateState) Task=`689eb1db-cb02-49a5-8d58-bff01b09eb91`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:10:22,709::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:10:22,710::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:10:22,710::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}} >Thread-14::DEBUG::2014-04-11 22:10:22,830::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4406499a] >Thread-14::DEBUG::2014-04-11 22:10:22,832::task::595::TaskManager.Task::(_updateState) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:10:22,832::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:10:22,833::task::866::TaskManager.Task::(_setError) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:10:22,834::task::885::TaskManager.Task::(_run) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::Task._run: 73e565a2-87cf-4511-a466-fb70604bd4be ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:10:22,834::task::1211::TaskManager.Task::(stop) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:10:22,835::task::990::TaskManager.Task::(_decref) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:10:22,835::task::1168::TaskManager.Task::(prepare) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:10:22,836::task::1173::TaskManager.Task::(prepare) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:10:22,836::task::990::TaskManager.Task::(_decref) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:10:22,837::task::925::TaskManager.Task::(_doAbort) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:10:22,837::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:10:22,838::task::595::TaskManager.Task::(_updateState) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:10:22,838::task::550::TaskManager.Task::(__state_aborting) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:10:22,839::task::595::TaskManager.Task::(_updateState) Task=`73e565a2-87cf-4511-a466-fb70604bd4be`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:10:22,839::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:10:22,840::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:10:22,840::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:10:22,884::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [4406499a] >Thread-14::DEBUG::2014-04-11 22:10:22,885::task::595::TaskManager.Task::(_updateState) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:10:22,885::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=36, domainsMap=None, options=None) >Thread-14::DEBUG::2014-04-11 22:10:22,886::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a51af771-9905-4e57-91c5-aba8fb1003a1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:10:22,887::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:10:22,887::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:10:22,888::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a51af771-9905-4e57-91c5-aba8fb1003a1`::Granted request >Thread-14::DEBUG::2014-04-11 22:10:22,889::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`4eed6604-b4da-47d0-80e4-7f81722b2ec1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:10:22,889::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:10:22,890::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:10:22,890::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`4eed6604-b4da-47d0-80e4-7f81722b2ec1`::Granted request >Thread-14::INFO::2014-04-11 22:10:22,891::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c0d8290 >Thread-14::INFO::2014-04-11 22:10:22,891::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 36) >Thread-14::DEBUG::2014-04-11 22:10:22,892::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:22,892::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:22,893::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:22,893::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:22,893::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:22,894::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:22,894::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:10:22,895::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:10:22,895::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 22:10:22,895::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:10:22,895::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 22:10:22,896::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 22:10:22,954::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:10:22,955::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:10:23,113::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:10:23,114::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:23,114::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:23,115::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:23,115::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:23,116::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:23,116::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:23,117::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::ERROR::2014-04-11 22:10:23,117::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::ERROR::2014-04-11 22:10:23,117::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:10:23,118::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:10:23,121::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-14::DEBUG::2014-04-11 22:10:23,478::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:10:23,481::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-14::DEBUG::2014-04-11 22:10:23,482::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:10:31,638::task::595::TaskManager.Task::(_updateState) Task=`c4e33ca2-e5c7-402b-bce5-7c8183a2dcd0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:10:31,639::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:10:31,639::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:10:31,639::task::1185::TaskManager.Task::(prepare) Task=`c4e33ca2-e5c7-402b-bce5-7c8183a2dcd0`::finished: {} >Thread-28::DEBUG::2014-04-11 22:10:31,639::task::595::TaskManager.Task::(_updateState) Task=`c4e33ca2-e5c7-402b-bce5-7c8183a2dcd0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:10:31,640::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:10:31,640::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:10:31,640::task::990::TaskManager.Task::(_decref) Task=`c4e33ca2-e5c7-402b-bce5-7c8183a2dcd0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:10:46,963::task::595::TaskManager.Task::(_updateState) Task=`0b23b7b8-52a8-49ef-b84e-25e1099cb64f`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:10:46,974::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:10:46,974::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:10:46,975::task::1185::TaskManager.Task::(prepare) Task=`0b23b7b8-52a8-49ef-b84e-25e1099cb64f`::finished: {} >Thread-28::DEBUG::2014-04-11 22:10:46,975::task::595::TaskManager.Task::(_updateState) Task=`0b23b7b8-52a8-49ef-b84e-25e1099cb64f`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:10:46,975::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:10:46,976::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:10:46,976::task::990::TaskManager.Task::(_decref) Task=`0b23b7b8-52a8-49ef-b84e-25e1099cb64f`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:11:02,273::task::595::TaskManager.Task::(_updateState) Task=`29121fff-2b45-4703-87ab-9160368a3c4b`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:11:02,274::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:11:02,274::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:11:02,275::task::1185::TaskManager.Task::(prepare) Task=`29121fff-2b45-4703-87ab-9160368a3c4b`::finished: {} >Thread-28::DEBUG::2014-04-11 22:11:02,275::task::595::TaskManager.Task::(_updateState) Task=`29121fff-2b45-4703-87ab-9160368a3c4b`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:11:02,275::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:11:02,276::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:11:02,276::task::990::TaskManager.Task::(_decref) Task=`29121fff-2b45-4703-87ab-9160368a3c4b`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:11:17,599::task::595::TaskManager.Task::(_updateState) Task=`206e204a-cb49-4902-8707-fa8b5b87d8f5`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:11:17,599::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:11:17,600::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:11:17,600::task::1185::TaskManager.Task::(prepare) Task=`206e204a-cb49-4902-8707-fa8b5b87d8f5`::finished: {} >Thread-28::DEBUG::2014-04-11 22:11:17,600::task::595::TaskManager.Task::(_updateState) Task=`206e204a-cb49-4902-8707-fa8b5b87d8f5`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:11:17,601::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:11:17,601::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:11:17,601::task::990::TaskManager.Task::(_decref) Task=`206e204a-cb49-4902-8707-fa8b5b87d8f5`::ref 0 aborting False >Thread-41::WARNING::2014-04-11 22:11:23,529::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-42::WARNING::2014-04-11 22:11:23,536::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-41::DEBUG::2014-04-11 22:11:23,549::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-43::WARNING::2014-04-11 22:11:23,540::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-42::DEBUG::2014-04-11 22:11:23,552::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-43::DEBUG::2014-04-11 22:11:23,557::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-14::ERROR::2014-04-11 22:11:23,560::sdc::143::Storage.StorageDomainCache::(_findDomain) domain a033c2ac-0d01-490c-9552-99ca53d6a64a not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',) >Thread-14::DEBUG::2014-04-11 22:11:23,561::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:11:23,562::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:11:23,562::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:11:23,563::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 22:11:23,563::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 22:11:23,564::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::DEBUG::2014-04-11 22:11:23,564::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:11:23,564::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-14::ERROR::2014-04-11 22:11:23,565::task::866::TaskManager.Task::(_setError) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool > spUUID, hostID, msdUUID, masterVersion, domainsMap) > File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool > res = pool.connect(hostID, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 625, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild > self.setMasterDomain(msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a' >Thread-14::DEBUG::2014-04-11 22:11:23,566::task::885::TaskManager.Task::(_run) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::Task._run: 3eb6aee2-6b1b-48b1-b8ee-120fcae16b90 ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 36, None) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:11:23,566::task::1211::TaskManager.Task::(stop) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:11:23,566::task::990::TaskManager.Task::(_decref) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:11:23,567::task::1168::TaskManager.Task::(prepare) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-14::DEBUG::2014-04-11 22:11:23,567::task::1173::TaskManager.Task::(prepare) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::Prepare: aborted: Cannot find master domain >Thread-14::DEBUG::2014-04-11 22:11:23,567::task::990::TaskManager.Task::(_decref) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:11:23,568::task::925::TaskManager.Task::(_doAbort) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:11:23,568::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:11:23,568::task::595::TaskManager.Task::(_updateState) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:11:23,569::task::550::TaskManager.Task::(__state_aborting) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:11:23,569::task::595::TaskManager.Task::(_updateState) Task=`3eb6aee2-6b1b-48b1-b8ee-120fcae16b90`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:11:23,569::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:11:23,570::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:11:23,570::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a'", 'code': 304}} >Thread-28::DEBUG::2014-04-11 22:11:23,803::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [200912ce] >Thread-28::DEBUG::2014-04-11 22:11:23,808::task::595::TaskManager.Task::(_updateState) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:11:23,809::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-28::ERROR::2014-04-11 22:11:23,809::task::866::TaskManager.Task::(_setError) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses > raise se.SpmStatusError() >SpmStatusError: Not SPM: () >Thread-28::DEBUG::2014-04-11 22:11:23,810::task::885::TaskManager.Task::(_run) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::Task._run: e28a0121-1ce3-4899-b539-ea0280711dbb () {} failed - stopping task >Thread-28::DEBUG::2014-04-11 22:11:23,810::task::1211::TaskManager.Task::(stop) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::stopping in state preparing (force False) >Thread-28::DEBUG::2014-04-11 22:11:23,810::task::990::TaskManager.Task::(_decref) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::ref 1 aborting True >Thread-28::INFO::2014-04-11 22:11:23,811::task::1168::TaskManager.Task::(prepare) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::aborting: Task is aborted: 'Not SPM' - code 654 >Thread-28::DEBUG::2014-04-11 22:11:23,811::task::1173::TaskManager.Task::(prepare) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::Prepare: aborted: Not SPM >Thread-28::DEBUG::2014-04-11 22:11:23,811::task::990::TaskManager.Task::(_decref) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::ref 0 aborting True >Thread-28::DEBUG::2014-04-11 22:11:23,812::task::925::TaskManager.Task::(_doAbort) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::Task._doAbort: force False >Thread-28::DEBUG::2014-04-11 22:11:23,812::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:11:23,812::task::595::TaskManager.Task::(_updateState) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::moving from state preparing -> state aborting >Thread-28::DEBUG::2014-04-11 22:11:23,813::task::550::TaskManager.Task::(__state_aborting) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::_aborting: recover policy none >Thread-28::DEBUG::2014-04-11 22:11:23,813::task::595::TaskManager.Task::(_updateState) Task=`e28a0121-1ce3-4899-b539-ea0280711dbb`::moving from state aborting -> state failed >Thread-28::DEBUG::2014-04-11 22:11:23,813::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:11:23,814::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::ERROR::2014-04-11 22:11:23,814::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}} >Thread-28::DEBUG::2014-04-11 22:11:23,908::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [200912ce] >Thread-28::DEBUG::2014-04-11 22:11:23,909::task::595::TaskManager.Task::(_updateState) Task=`7fa4df86-db61-49a0-8afe-1b35d290fb8b`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:11:23,909::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, remove=False, options=None) >Thread-28::WARNING::2014-04-11 22:11:23,910::hsm::1116::Storage.HSM::(disconnectStoragePool) disconnect sp: a106ab81-9d5f-49c1-aeaf-832a137b708c failed. Known pools {} >Thread-28::INFO::2014-04-11 22:11:23,910::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None >Thread-28::DEBUG::2014-04-11 22:11:23,910::task::1185::TaskManager.Task::(prepare) Task=`7fa4df86-db61-49a0-8afe-1b35d290fb8b`::finished: None >Thread-28::DEBUG::2014-04-11 22:11:23,911::task::595::TaskManager.Task::(_updateState) Task=`7fa4df86-db61-49a0-8afe-1b35d290fb8b`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:11:23,911::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:11:23,911::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:11:23,912::task::990::TaskManager.Task::(_decref) Task=`7fa4df86-db61-49a0-8afe-1b35d290fb8b`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:11:23,956::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [200912ce] >Thread-28::DEBUG::2014-04-11 22:11:23,957::task::595::TaskManager.Task::(_updateState) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:11:23,958::logUtils::44::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', poolName='SuprTEK', masterDom='3406665e-4adc-4fd4-aa1e-037547b29adb', domDict={'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'active', '21619c8e-99ea-4813-be02-d708971e5393': 'active'}, masterVersion=37, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=1, options=None) >Thread-28::INFO::2014-04-11 22:11:23,958::hsm::1852::Storage.HSM::(reconstructMaster) spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c master=3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-28::INFO::2014-04-11 22:11:23,958::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e9c125830 >Thread-28::ERROR::2014-04-11 22:11:23,959::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-28::ERROR::2014-04-11 22:11:23,959::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-28::DEBUG::2014-04-11 22:11:23,960::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:11:23,961::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-28::DEBUG::2014-04-11 22:11:24,325::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-28::WARNING::2014-04-11 22:11:24,330::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-28::DEBUG::2014-04-11 22:11:24,331::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:11:30,400::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [2e43b400] >Thread-14::DEBUG::2014-04-11 22:11:30,402::task::595::TaskManager.Task::(_updateState) Task=`b8db45e8-e128-44b3-8f29-1914681f9624`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:11:30,402::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'ashtistg01.suprtekstic.com:/storage/infrastructure', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e', 'port': ''}], options=None) >Thread-14::DEBUG::2014-04-11 22:11:30,415::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/infrastructure /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure' (cwd None) >Thread-14::ERROR::2014-04-11 22:12:22,787::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect > self._mount.mount(self.options, self._vfsType) > File "/usr/share/vdsm/storage/mount.py", line 222, in mount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-14::ERROR::2014-04-11 22:12:22,792::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer > conObj.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect > return self._mountCon.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect > raise e >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-14::DEBUG::2014-04-11 22:12:22,794::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain} >Thread-14::INFO::2014-04-11 22:12:22,796::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}]} >Thread-14::DEBUG::2014-04-11 22:12:22,797::task::1185::TaskManager.Task::(prepare) Task=`b8db45e8-e128-44b3-8f29-1914681f9624`::finished: {'statuslist': [{'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}]} >Thread-14::DEBUG::2014-04-11 22:12:22,797::task::595::TaskManager.Task::(_updateState) Task=`b8db45e8-e128-44b3-8f29-1914681f9624`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:12:22,799::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:12:22,800::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:12:22,801::task::990::TaskManager.Task::(_decref) Task=`b8db45e8-e128-44b3-8f29-1914681f9624`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:12:22,845::task::595::TaskManager.Task::(_updateState) Task=`973611d7-8fe8-4801-9ad8-ec9776b694fd`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:12:22,853::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:12:22,860::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-14::DEBUG::2014-04-11 22:12:22,865::task::1185::TaskManager.Task::(prepare) Task=`973611d7-8fe8-4801-9ad8-ec9776b694fd`::finished: {} >Thread-14::DEBUG::2014-04-11 22:12:22,872::task::595::TaskManager.Task::(_updateState) Task=`973611d7-8fe8-4801-9ad8-ec9776b694fd`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:12:22,876::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:12:22,880::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:12:22,886::task::990::TaskManager.Task::(_decref) Task=`973611d7-8fe8-4801-9ad8-ec9776b694fd`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:12:22,923::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6d750a9d] >Thread-14::DEBUG::2014-04-11 22:12:22,925::task::595::TaskManager.Task::(_updateState) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:12:22,926::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:12:22,927::task::866::TaskManager.Task::(_setError) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:12:22,929::task::885::TaskManager.Task::(_run) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::Task._run: 08c5c6da-1f8d-4add-bbee-f42ac0831116 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:12:22,930::task::1211::TaskManager.Task::(stop) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:12:22,931::task::990::TaskManager.Task::(_decref) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:12:22,932::task::1168::TaskManager.Task::(prepare) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:12:22,933::task::1173::TaskManager.Task::(prepare) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:12:22,934::task::990::TaskManager.Task::(_decref) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:12:22,934::task::925::TaskManager.Task::(_doAbort) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:12:22,935::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:12:22,936::task::595::TaskManager.Task::(_updateState) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:12:22,937::task::550::TaskManager.Task::(__state_aborting) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:12:22,937::task::595::TaskManager.Task::(_updateState) Task=`08c5c6da-1f8d-4add-bbee-f42ac0831116`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:12:22,938::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:12:22,940::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:12:22,941::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:12:23,033::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [6d750a9d] >Thread-14::DEBUG::2014-04-11 22:12:23,035::task::595::TaskManager.Task::(_updateState) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:12:23,035::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='3406665e-4adc-4fd4-aa1e-037547b29adb', masterVersion=37, domainsMap=None, options=None) >Thread-14::DEBUG::2014-04-11 22:12:23,039::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e9c3b498-50aa-4458-8c92-0b52e1d9e92d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:12:23,039::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:12:23,041::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:12:23,041::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e9c3b498-50aa-4458-8c92-0b52e1d9e92d`::Granted request >Thread-14::DEBUG::2014-04-11 22:12:23,045::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`3e1d779c-fad3-412d-bc1c-9eddde0d9126`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:12:23,046::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:12:23,047::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:12:23,048::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`3e1d779c-fad3-412d-bc1c-9eddde0d9126`::Granted request >Thread-14::INFO::2014-04-11 22:12:23,050::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e8c075368 >Thread-14::INFO::2014-04-11 22:12:23,050::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: 3406665e-4adc-4fd4-aa1e-037547b29adb (ver = 37) >Thread-14::DEBUG::2014-04-11 22:12:23,052::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:23,053::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:23,055::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:23,056::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:23,057::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:23,059::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-47::WARNING::2014-04-11 22:12:24,387::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-47::DEBUG::2014-04-11 22:12:24,399::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-48::WARNING::2014-04-11 22:12:24,399::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-49::WARNING::2014-04-11 22:12:24,402::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-48::DEBUG::2014-04-11 22:12:24,408::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-49::DEBUG::2014-04-11 22:12:24,409::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-28::ERROR::2014-04-11 22:12:24,412::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:12:24,413::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:12:24,415::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-28::ERROR::2014-04-11 22:12:24,413::task::866::TaskManager.Task::(_setError) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1862, in reconstructMaster > self.validateSdUUID(masterDom) > File "/usr/share/vdsm/storage/hsm.py", line 270, in validateSdUUID > sdDom = sdCache.produce(sdUUID=sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:12:24,417::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-28::DEBUG::2014-04-11 22:12:24,418::task::885::TaskManager.Task::(_run) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::Task._run: f4e11904-c42d-46f9-b869-dfcf9616fca1 ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 'SuprTEK', '3406665e-4adc-4fd4-aa1e-037547b29adb', {'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'active', '21619c8e-99ea-4813-be02-d708971e5393': 'active'}, 37, None, 5, 60, 10, 3, 1) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:12:24,419::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-28::DEBUG::2014-04-11 22:12:24,421::task::1211::TaskManager.Task::(stop) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:12:24,422::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-28::DEBUG::2014-04-11 22:12:24,424::task::990::TaskManager.Task::(_decref) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::ref 1 aborting True >Thread-14::DEBUG::2014-04-11 22:12:24,425::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-28::INFO::2014-04-11 22:12:24,426::task::1168::TaskManager.Task::(prepare) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 >Thread-28::DEBUG::2014-04-11 22:12:24,437::task::1173::TaskManager.Task::(prepare) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::Prepare: aborted: Storage domain does not exist >Thread-28::DEBUG::2014-04-11 22:12:24,438::task::990::TaskManager.Task::(_decref) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::ref 0 aborting True >Thread-28::DEBUG::2014-04-11 22:12:24,438::task::925::TaskManager.Task::(_doAbort) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::Task._doAbort: force False >Thread-28::DEBUG::2014-04-11 22:12:24,439::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:12:24,439::task::595::TaskManager.Task::(_updateState) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::moving from state preparing -> state aborting >Thread-28::DEBUG::2014-04-11 22:12:24,440::task::550::TaskManager.Task::(__state_aborting) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::_aborting: recover policy none >Thread-28::DEBUG::2014-04-11 22:12:24,440::task::595::TaskManager.Task::(_updateState) Task=`f4e11904-c42d-46f9-b869-dfcf9616fca1`::moving from state aborting -> state failed >Thread-28::DEBUG::2014-04-11 22:12:24,441::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:12:24,441::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::ERROR::2014-04-11 22:12:24,442::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',)", 'code': 358}} >Thread-14::DEBUG::2014-04-11 22:12:24,458::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:12:24,458::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:12:24,612::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:12:24,613::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:24,614::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:24,614::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:24,615::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:24,615::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:24,616::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:24,616::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::ERROR::2014-04-11 22:12:24,616::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-14::ERROR::2014-04-11 22:12:24,617::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-14::DEBUG::2014-04-11 22:12:24,617::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:12:24,620::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 22:12:24,991::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:12:24,993::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-14::DEBUG::2014-04-11 22:12:24,993::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:12:38,278::task::595::TaskManager.Task::(_updateState) Task=`236b548a-14d8-4b8d-b9ea-aebe54b6adc3`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:12:38,280::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:12:38,280::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:12:38,280::task::1185::TaskManager.Task::(prepare) Task=`236b548a-14d8-4b8d-b9ea-aebe54b6adc3`::finished: {} >Thread-28::DEBUG::2014-04-11 22:12:38,281::task::595::TaskManager.Task::(_updateState) Task=`236b548a-14d8-4b8d-b9ea-aebe54b6adc3`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:12:38,281::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:12:38,281::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:12:38,282::task::990::TaskManager.Task::(_decref) Task=`236b548a-14d8-4b8d-b9ea-aebe54b6adc3`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:12:53,589::task::595::TaskManager.Task::(_updateState) Task=`60bcdbbf-1636-4ef3-8537-c6998b6421e1`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:12:53,590::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:12:53,590::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:12:53,590::task::1185::TaskManager.Task::(prepare) Task=`60bcdbbf-1636-4ef3-8537-c6998b6421e1`::finished: {} >Thread-28::DEBUG::2014-04-11 22:12:53,591::task::595::TaskManager.Task::(_updateState) Task=`60bcdbbf-1636-4ef3-8537-c6998b6421e1`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:12:53,591::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:12:53,591::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:12:53,592::task::990::TaskManager.Task::(_decref) Task=`60bcdbbf-1636-4ef3-8537-c6998b6421e1`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:13:08,901::task::595::TaskManager.Task::(_updateState) Task=`bc5b9679-c0ca-490b-8a2d-b1be06ca5bd0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:13:08,901::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:13:08,902::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:13:08,902::task::1185::TaskManager.Task::(prepare) Task=`bc5b9679-c0ca-490b-8a2d-b1be06ca5bd0`::finished: {} >Thread-28::DEBUG::2014-04-11 22:13:08,902::task::595::TaskManager.Task::(_updateState) Task=`bc5b9679-c0ca-490b-8a2d-b1be06ca5bd0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:13:08,903::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:13:08,903::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:13:08,903::task::990::TaskManager.Task::(_decref) Task=`bc5b9679-c0ca-490b-8a2d-b1be06ca5bd0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:13:24,233::task::595::TaskManager.Task::(_updateState) Task=`4071e668-41a8-4d03-b074-87c8d51e6d80`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:13:24,236::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28::INFO::2014-04-11 22:13:24,238::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-28::DEBUG::2014-04-11 22:13:24,239::task::1185::TaskManager.Task::(prepare) Task=`4071e668-41a8-4d03-b074-87c8d51e6d80`::finished: {} >Thread-28::DEBUG::2014-04-11 22:13:24,239::task::595::TaskManager.Task::(_updateState) Task=`4071e668-41a8-4d03-b074-87c8d51e6d80`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:13:24,240::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:13:24,241::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:13:24,243::task::990::TaskManager.Task::(_decref) Task=`4071e668-41a8-4d03-b074-87c8d51e6d80`::ref 0 aborting False >Thread-54::WARNING::2014-04-11 22:13:25,049::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-55::WARNING::2014-04-11 22:13:25,051::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-54::DEBUG::2014-04-11 22:13:25,064::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-55::DEBUG::2014-04-11 22:13:25,067::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-14::ERROR::2014-04-11 22:13:25,070::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:13:25,071::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:13:25,071::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:13:25,071::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:13:25,072::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 22:13:25,072::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 22:13:25,073::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::DEBUG::2014-04-11 22:13:25,073::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:13:25,073::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-14::ERROR::2014-04-11 22:13:25,074::task::866::TaskManager.Task::(_setError) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool > spUUID, hostID, msdUUID, masterVersion, domainsMap) > File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool > res = pool.connect(hostID, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 625, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild > self.setMasterDomain(msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1351, in setMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=3406665e-4adc-4fd4-aa1e-037547b29adb' >Thread-14::DEBUG::2014-04-11 22:13:25,075::task::885::TaskManager.Task::(_run) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::Task._run: f4279d64-2d14-44e0-a81c-131804e3f4f7 ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, '3406665e-4adc-4fd4-aa1e-037547b29adb', 37, None) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:13:25,075::task::1211::TaskManager.Task::(stop) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:13:25,076::task::990::TaskManager.Task::(_decref) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:13:25,076::task::1168::TaskManager.Task::(prepare) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-14::DEBUG::2014-04-11 22:13:25,076::task::1173::TaskManager.Task::(prepare) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::Prepare: aborted: Cannot find master domain >Thread-14::DEBUG::2014-04-11 22:13:25,077::task::990::TaskManager.Task::(_decref) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:13:25,077::task::925::TaskManager.Task::(_doAbort) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:13:25,077::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:13:25,078::task::595::TaskManager.Task::(_updateState) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:13:25,078::task::550::TaskManager.Task::(__state_aborting) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:13:25,078::task::595::TaskManager.Task::(_updateState) Task=`f4279d64-2d14-44e0-a81c-131804e3f4f7`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:13:25,079::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:13:25,079::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:13:25,079::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c, msdUUID=3406665e-4adc-4fd4-aa1e-037547b29adb'", 'code': 304}} >Thread-14::DEBUG::2014-04-11 22:13:25,282::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [29eaa508] >Thread-14::DEBUG::2014-04-11 22:13:25,283::task::595::TaskManager.Task::(_updateState) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:13:25,284::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:13:25,284::task::866::TaskManager.Task::(_setError) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-14::DEBUG::2014-04-11 22:13:25,285::task::885::TaskManager.Task::(_run) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::Task._run: 61093c90-ac6a-475d-98a0-c14276df11e1 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:13:25,285::task::1211::TaskManager.Task::(stop) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:13:25,285::task::990::TaskManager.Task::(_decref) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:13:25,286::task::1168::TaskManager.Task::(prepare) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-14::DEBUG::2014-04-11 22:13:25,286::task::1173::TaskManager.Task::(prepare) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::Prepare: aborted: Unknown pool id, pool not connected >Thread-14::DEBUG::2014-04-11 22:13:25,286::task::990::TaskManager.Task::(_decref) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:13:25,287::task::925::TaskManager.Task::(_doAbort) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:13:25,287::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:13:25,287::task::595::TaskManager.Task::(_updateState) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:13:25,288::task::550::TaskManager.Task::(__state_aborting) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:13:25,288::task::595::TaskManager.Task::(_updateState) Task=`61093c90-ac6a-475d-98a0-c14276df11e1`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:13:25,288::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:13:25,289::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:13:25,289::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-14::DEBUG::2014-04-11 22:13:25,336::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [29eaa508] >Thread-14::DEBUG::2014-04-11 22:13:25,337::task::595::TaskManager.Task::(_updateState) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:13:25,338::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=37, domainsMap=None, options=None) >Thread-14::DEBUG::2014-04-11 22:13:25,338::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`3e8aafa4-01c2-4301-aa64-5c10076bf1c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:13:25,339::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:13:25,339::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:13:25,340::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`3e8aafa4-01c2-4301-aa64-5c10076bf1c2`::Granted request >Thread-14::DEBUG::2014-04-11 22:13:25,341::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`7ffc5d06-6de3-4202-a871-44660affb354`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:13:25,341::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:13:25,341::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:13:25,342::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`7ffc5d06-6de3-4202-a871-44660affb354`::Granted request >Thread-14::INFO::2014-04-11 22:13:25,343::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e840890e0 >Thread-14::INFO::2014-04-11 22:13:25,343::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 37) >Thread-14::DEBUG::2014-04-11 22:13:25,343::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,344::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,344::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,345::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,345::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,346::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,346::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:13:25,346::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:13:25,347::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 22:13:25,347::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:13:25,347::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 22:13:25,348::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 22:13:25,393::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:13:25,394::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:13:25,534::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:13:25,535::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,535::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,536::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,536::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,537::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,537::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,537::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::ERROR::2014-04-11 22:13:25,538::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::ERROR::2014-04-11 22:13:25,538::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:13:25,539::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,542::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-14::DEBUG::2014-04-11 22:13:25,892::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:13:25,894::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-14::DEBUG::2014-04-11 22:13:25,894::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:25,972::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:13:26,246::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-14::DEBUG::2014-04-11 22:13:26,264::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=36', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a76d9a8af2ad818a809eee527dd4e251532a84c3'] >Thread-14::DEBUG::2014-04-11 22:13:26,267::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-14::DEBUG::2014-04-11 22:13:26,268::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS' >Thread-14::DEBUG::2014-04-11 22:13:26,269::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS' >Thread-14::ERROR::2014-04-11 22:13:26,270::spbackends::282::Storage.StoragePoolDiskBackend::(validateMasterDomainVersion) Requested master domain a033c2ac-0d01-490c-9552-99ca53d6a64a does not have expected version 37 it is version 36 >Thread-14::DEBUG::2014-04-11 22:13:26,270::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:13:26,271::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:13:26,271::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:13:26,271::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 22:13:26,272::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 22:13:26,272::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::DEBUG::2014-04-11 22:13:26,273::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:13:26,273::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-14::ERROR::2014-04-11 22:13:26,273::task::866::TaskManager.Task::(_setError) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1010, in connectStoragePool > spUUID, hostID, msdUUID, masterVersion, domainsMap) > File "/usr/share/vdsm/storage/hsm.py", line 1081, in _connectStoragePool > res = pool.connect(hostID, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 625, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1144, in __rebuild > self.setMasterDomain(msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1364, in setMasterDomain > self._backend.validateMasterDomainVersion(domain, masterVersion) > File "/usr/share/vdsm/storage/spbackends.py", line 283, in validateMasterDomainVersion > raise se.StoragePoolWrongMaster(self.spUUID, masterDomain.sdUUID) >StoragePoolWrongMaster: Wrong Master domain or its version: 'SD=a033c2ac-0d01-490c-9552-99ca53d6a64a, pool=a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:13:26,283::task::885::TaskManager.Task::(_run) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::Task._run: feb8d088-f530-4b72-8cb1-f46ab75a6ddf ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, 'a033c2ac-0d01-490c-9552-99ca53d6a64a', 37, None) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:13:26,283::task::1211::TaskManager.Task::(stop) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:13:26,283::task::990::TaskManager.Task::(_decref) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:13:26,284::task::1168::TaskManager.Task::(prepare) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::aborting: Task is aborted: 'Wrong Master domain or its version' - code 324 >Thread-14::DEBUG::2014-04-11 22:13:26,284::task::1173::TaskManager.Task::(prepare) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::Prepare: aborted: Wrong Master domain or its version >Thread-14::DEBUG::2014-04-11 22:13:26,285::task::990::TaskManager.Task::(_decref) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:13:26,285::task::925::TaskManager.Task::(_doAbort) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:13:26,285::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:13:26,286::task::595::TaskManager.Task::(_updateState) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:13:26,286::task::550::TaskManager.Task::(__state_aborting) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:13:26,286::task::595::TaskManager.Task::(_updateState) Task=`feb8d088-f530-4b72-8cb1-f46ab75a6ddf`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:13:26,287::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:13:26,287::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:13:26,287::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Wrong Master domain or its version: 'SD=a033c2ac-0d01-490c-9552-99ca53d6a64a, pool=a106ab81-9d5f-49c1-aeaf-832a137b708c'", 'code': 324}} >Thread-14::DEBUG::2014-04-11 22:13:26,437::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [5f90a775] >Thread-14::DEBUG::2014-04-11 22:13:26,438::task::595::TaskManager.Task::(_updateState) Task=`79f16959-34cd-4c38-bd5b-07fc4634baba`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:13:26,439::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, remove=False, options=None) >Thread-14::WARNING::2014-04-11 22:13:26,439::hsm::1116::Storage.HSM::(disconnectStoragePool) disconnect sp: a106ab81-9d5f-49c1-aeaf-832a137b708c failed. Known pools {} >Thread-14::INFO::2014-04-11 22:13:26,439::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None >Thread-14::DEBUG::2014-04-11 22:13:26,440::task::1185::TaskManager.Task::(prepare) Task=`79f16959-34cd-4c38-bd5b-07fc4634baba`::finished: None >Thread-14::DEBUG::2014-04-11 22:13:26,440::task::595::TaskManager.Task::(_updateState) Task=`79f16959-34cd-4c38-bd5b-07fc4634baba`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:13:26,440::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:13:26,441::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:13:26,441::task::990::TaskManager.Task::(_decref) Task=`79f16959-34cd-4c38-bd5b-07fc4634baba`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:13:26,484::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [5f90a775] >Thread-14::DEBUG::2014-04-11 22:13:26,485::task::595::TaskManager.Task::(_updateState) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:13:26,486::logUtils::44::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', poolName='SuprTEK', masterDom='a033c2ac-0d01-490c-9552-99ca53d6a64a', domDict={'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'active', '21619c8e-99ea-4813-be02-d708971e5393': 'active'}, masterVersion=38, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=1, options=None) >Thread-14::INFO::2014-04-11 22:13:26,486::hsm::1852::Storage.HSM::(reconstructMaster) spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c master=a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::INFO::2014-04-11 22:13:26,487::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e9c110dd0 >Thread-14::INFO::2014-04-11 22:13:26,487::fileSD::316::Storage.StorageDomain::(validate) sdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:13:26,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=36', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a76d9a8af2ad818a809eee527dd4e251532a84c3'] >Thread-14::DEBUG::2014-04-11 22:13:26,505::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`cb7b76ab-6035-405f-bb8a-721329082378`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1867' at 'reconstructMaster' >Thread-14::DEBUG::2014-04-11 22:13:26,506::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:13:26,506::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:13:26,506::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`cb7b76ab-6035-405f-bb8a-721329082378`::Granted request >Thread-14::DEBUG::2014-04-11 22:13:26,507::task::827::TaskManager.Task::(resourceAcquired) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive) >Thread-14::DEBUG::2014-04-11 22:13:26,508::task::990::TaskManager.Task::(_decref) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::ref 1 aborting False >Thread-14::INFO::2014-04-11 22:13:26,508::sp::679::Storage.StoragePool::(reconstructMaster) spUUID=a106ab81-9d5f-49c1-aeaf-832a137b708c hostId=1 poolName=SuprTEK msdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a domDict={'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'active', '21619c8e-99ea-4813-be02-d708971e5393': 'active'} masterVersion=38 leaseparams=({'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5}) >Thread-14::INFO::2014-04-11 22:13:26,508::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-28::DEBUG::2014-04-11 22:13:35,210::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:13:35,211::task::595::TaskManager.Task::(_updateState) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:13:35,212::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-28::ERROR::2014-04-11 22:13:35,212::task::866::TaskManager.Task::(_setError) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-28::DEBUG::2014-04-11 22:13:35,213::task::885::TaskManager.Task::(_run) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::Task._run: 7c9bf435-18ec-4e04-b2a2-9032d70a5669 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-28::DEBUG::2014-04-11 22:13:35,214::task::1211::TaskManager.Task::(stop) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::stopping in state preparing (force False) >Thread-28::DEBUG::2014-04-11 22:13:35,214::task::990::TaskManager.Task::(_decref) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::ref 1 aborting True >Thread-28::INFO::2014-04-11 22:13:35,214::task::1168::TaskManager.Task::(prepare) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-28::DEBUG::2014-04-11 22:13:35,214::task::1173::TaskManager.Task::(prepare) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::Prepare: aborted: Unknown pool id, pool not connected >Thread-28::DEBUG::2014-04-11 22:13:35,215::task::990::TaskManager.Task::(_decref) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::ref 0 aborting True >Thread-28::DEBUG::2014-04-11 22:13:35,215::task::925::TaskManager.Task::(_doAbort) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::Task._doAbort: force False >Thread-28::DEBUG::2014-04-11 22:13:35,215::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:13:35,216::task::595::TaskManager.Task::(_updateState) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::moving from state preparing -> state aborting >Thread-28::DEBUG::2014-04-11 22:13:35,216::task::550::TaskManager.Task::(__state_aborting) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::_aborting: recover policy none >Thread-28::DEBUG::2014-04-11 22:13:35,216::task::595::TaskManager.Task::(_updateState) Task=`7c9bf435-18ec-4e04-b2a2-9032d70a5669`::moving from state aborting -> state failed >Thread-28::DEBUG::2014-04-11 22:13:35,216::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:13:35,217::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::ERROR::2014-04-11 22:13:35,217::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-28::DEBUG::2014-04-11 22:13:35,221::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:13:35,222::task::595::TaskManager.Task::(_updateState) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:13:35,222::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-28::ERROR::2014-04-11 22:13:35,222::task::866::TaskManager.Task::(_setError) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses > raise se.SpmStatusError() >SpmStatusError: Not SPM: () >Thread-28::DEBUG::2014-04-11 22:13:35,223::task::885::TaskManager.Task::(_run) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::Task._run: ddcfbb3b-268f-4fff-ae92-a8a40beef2e2 () {} failed - stopping task >Thread-28::DEBUG::2014-04-11 22:13:35,223::task::1211::TaskManager.Task::(stop) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::stopping in state preparing (force False) >Thread-28::DEBUG::2014-04-11 22:13:35,224::task::990::TaskManager.Task::(_decref) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::ref 1 aborting True >Thread-28::INFO::2014-04-11 22:13:35,224::task::1168::TaskManager.Task::(prepare) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::aborting: Task is aborted: 'Not SPM' - code 654 >Thread-28::DEBUG::2014-04-11 22:13:35,224::task::1173::TaskManager.Task::(prepare) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::Prepare: aborted: Not SPM >Thread-28::DEBUG::2014-04-11 22:13:35,224::task::990::TaskManager.Task::(_decref) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::ref 0 aborting True >Thread-28::DEBUG::2014-04-11 22:13:35,225::task::925::TaskManager.Task::(_doAbort) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::Task._doAbort: force False >Thread-28::DEBUG::2014-04-11 22:13:35,225::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:13:35,225::task::595::TaskManager.Task::(_updateState) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::moving from state preparing -> state aborting >Thread-28::DEBUG::2014-04-11 22:13:35,226::task::550::TaskManager.Task::(__state_aborting) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::_aborting: recover policy none >Thread-28::DEBUG::2014-04-11 22:13:35,226::task::595::TaskManager.Task::(_updateState) Task=`ddcfbb3b-268f-4fff-ae92-a8a40beef2e2`::moving from state aborting -> state failed >Thread-28::DEBUG::2014-04-11 22:13:35,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:13:35,226::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::ERROR::2014-04-11 22:13:35,227::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}} >Thread-28::DEBUG::2014-04-11 22:13:35,343::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:13:35,344::task::595::TaskManager.Task::(_updateState) Task=`be16651a-0402-420a-b0d6-08f782484585`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:13:35,344::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-28::ERROR::2014-04-11 22:13:35,345::task::866::TaskManager.Task::(_setError) Task=`be16651a-0402-420a-b0d6-08f782484585`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 619, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 325, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-28::DEBUG::2014-04-11 22:13:35,345::task::885::TaskManager.Task::(_run) Task=`be16651a-0402-420a-b0d6-08f782484585`::Task._run: be16651a-0402-420a-b0d6-08f782484585 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-28::DEBUG::2014-04-11 22:13:35,346::task::1211::TaskManager.Task::(stop) Task=`be16651a-0402-420a-b0d6-08f782484585`::stopping in state preparing (force False) >Thread-28::DEBUG::2014-04-11 22:13:35,346::task::990::TaskManager.Task::(_decref) Task=`be16651a-0402-420a-b0d6-08f782484585`::ref 1 aborting True >Thread-28::INFO::2014-04-11 22:13:35,346::task::1168::TaskManager.Task::(prepare) Task=`be16651a-0402-420a-b0d6-08f782484585`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-28::DEBUG::2014-04-11 22:13:35,347::task::1173::TaskManager.Task::(prepare) Task=`be16651a-0402-420a-b0d6-08f782484585`::Prepare: aborted: Unknown pool id, pool not connected >Thread-28::DEBUG::2014-04-11 22:13:35,347::task::990::TaskManager.Task::(_decref) Task=`be16651a-0402-420a-b0d6-08f782484585`::ref 0 aborting True >Thread-28::DEBUG::2014-04-11 22:13:35,347::task::925::TaskManager.Task::(_doAbort) Task=`be16651a-0402-420a-b0d6-08f782484585`::Task._doAbort: force False >Thread-28::DEBUG::2014-04-11 22:13:35,348::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:13:35,348::task::595::TaskManager.Task::(_updateState) Task=`be16651a-0402-420a-b0d6-08f782484585`::moving from state preparing -> state aborting >Thread-28::DEBUG::2014-04-11 22:13:35,348::task::550::TaskManager.Task::(__state_aborting) Task=`be16651a-0402-420a-b0d6-08f782484585`::_aborting: recover policy none >Thread-28::DEBUG::2014-04-11 22:13:35,349::task::595::TaskManager.Task::(_updateState) Task=`be16651a-0402-420a-b0d6-08f782484585`::moving from state aborting -> state failed >Thread-28::DEBUG::2014-04-11 22:13:35,349::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:13:35,349::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::ERROR::2014-04-11 22:13:35,350::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 309}} >Thread-28::DEBUG::2014-04-11 22:13:35,393::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:13:35,394::task::595::TaskManager.Task::(_updateState) Task=`bb2d61f2-9971-4bcc-9374-5e2746c7e69f`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:13:35,394::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=38, domainsMap=None, options=None) >Thread-28::DEBUG::2014-04-11 22:13:35,395::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`5bf74bd3-dade-404d-bcae-4a5dfe36ef9f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-28::DEBUG::2014-04-11 22:13:35,396::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-28::DEBUG::2014-04-11 22:13:35,396::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-28::DEBUG::2014-04-11 22:13:35,396::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`5bf74bd3-dade-404d-bcae-4a5dfe36ef9f`::Granted request >Thread-28::DEBUG::2014-04-11 22:13:35,399::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`4f06bab6-01cb-4fbf-8307-932a9ee6318c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-28::DEBUG::2014-04-11 22:13:35,399::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-28::DEBUG::2014-04-11 22:13:35,400::resourceManager::577::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is currently locked, Entering queue (1 in queue) >Thread-14::DEBUG::2014-04-11 22:13:47,513::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >Thread-14::INFO::2014-04-11 22:13:47,525::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-14::DEBUG::2014-04-11 22:13:47,580::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >Thread-14::INFO::2014-04-11 22:13:47,581::sp::666::Storage.StoragePool::(createMaster) setting master domain for spUUID a106ab81-9d5f-49c1-aeaf-832a137b708c on sdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:13:47,581::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-14::DEBUG::2014-04-11 22:13:47,582::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-14::DEBUG::2014-04-11 22:13:47,582::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=a033c2ac-0d01-490c-9552-99ca53d6a64a:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2e7a13952d8abc1dc29bd0da89fde82273217f47'] >Thread-14::DEBUG::2014-04-11 22:13:47,611::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-14::DEBUG::2014-04-11 22:13:47,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=a033c2ac-0d01-490c-9552-99ca53d6a64a:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2e7a13952d8abc1dc29bd0da89fde82273217f47'] >Thread-14::DEBUG::2014-04-11 22:13:47,632::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-14::DEBUG::2014-04-11 22:13:47,632::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-14::DEBUG::2014-04-11 22:13:47,632::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-14::DEBUG::2014-04-11 22:13:47,633::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-14::DEBUG::2014-04-11 22:13:47,633::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-14::DEBUG::2014-04-11 22:13:47,633::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38 >Thread-14::DEBUG::2014-04-11 22:13:47,634::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-14::DEBUG::2014-04-11 22:13:47,634::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:13:47,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=a033c2ac-0d01-490c-9552-99ca53d6a64a:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2e7a13952d8abc1dc29bd0da89fde82273217f47'] >Thread-14::INFO::2014-04-11 22:13:47,650::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-65::DEBUG::2014-04-11 22:13:47,651::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:13:47,651::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `a033c2ac-0d01-490c-9552-99ca53d6a64a` >Thread-14::DEBUG::2014-04-11 22:13:47,652::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::INFO::2014-04-11 22:13:47,654::sp::716::Storage.StoragePool::(reconstructMaster) Set storage pool domains: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': 'Active', 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': 'Active', 'b7663d70-e658-41fa-b9f0-8da83c9eddce': 'Active', '3406665e-4adc-4fd4-aa1e-037547b29adb': 'Active', 'f3b51811-4a7f-43af-8633-322b3db23c48': 'Active', 'a033c2ac-0d01-490c-9552-99ca53d6a64a': 'Active', '21619c8e-99ea-4813-be02-d708971e5393': 'Active'} >Thread-14::DEBUG::2014-04-11 22:13:47,655::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-14::DEBUG::2014-04-11 22:13:47,656::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-14::DEBUG::2014-04-11 22:13:47,657::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-14::DEBUG::2014-04-11 22:13:47,673::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-14::DEBUG::2014-04-11 22:13:47,674::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,674::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,675::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,675::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,675::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,676::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,676::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:13:47,676::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:13:47,677::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 22:13:47,677::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:13:47,677::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 22:13:47,678::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 22:13:47,728::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:13:47,729::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:13:47,851::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:13:47,852::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,853::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,853::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,854::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,854::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,855::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,855::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::ERROR::2014-04-11 22:13:47,855::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::ERROR::2014-04-11 22:13:47,856::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:13:47,856::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:47,862::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-14::DEBUG::2014-04-11 22:13:48,198::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:13:48,201::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-14::DEBUG::2014-04-11 22:13:48,202::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:13:48,273::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:13:48,276::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-14::DEBUG::2014-04-11 22:13:48,293::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-14::DEBUG::2014-04-11 22:13:48,296::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-14::INFO::2014-04-11 22:13:48,296::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered >Thread-14::INFO::2014-04-11 22:13:48,297::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered >Thread-14::DEBUG::2014-04-11 22:13:48,299::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38 >Thread-14::DEBUG::2014-04-11 22:13:48,301::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-14::DEBUG::2014-04-11 22:13:48,304::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:13:48,330::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-65::DEBUG::2014-04-11 22:13:48,331::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-14::INFO::2014-04-11 22:13:48,346::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-72::DEBUG::2014-04-11 22:13:48,350::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-14::DEBUG::2014-04-11 22:13:48,350::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `423b56b3-3148-468c-8c28-4c96fddbc1c8` >Thread-72::ERROR::2014-04-11 22:13:48,355::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-14::INFO::2014-04-11 22:13:48,356::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-72::ERROR::2014-04-11 22:13:48,359::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-73::DEBUG::2014-04-11 22:13:48,366::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-14::DEBUG::2014-04-11 22:13:48,365::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` >Thread-72::DEBUG::2014-04-11 22:13:48,369::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::INFO::2014-04-11 22:13:48,373::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-74::DEBUG::2014-04-11 22:13:48,388::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-14::DEBUG::2014-04-11 22:13:48,387::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `b7663d70-e658-41fa-b9f0-8da83c9eddce` >Thread-74::ERROR::2014-04-11 22:13:48,403::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-14::INFO::2014-04-11 22:13:48,403::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-74::ERROR::2014-04-11 22:13:48,442::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-14::DEBUG::2014-04-11 22:13:48,465::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `3406665e-4adc-4fd4-aa1e-037547b29adb` >Thread-81::DEBUG::2014-04-11 22:13:48,466::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-74::DEBUG::2014-04-11 22:13:48,474::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-65::DEBUG::2014-04-11 22:13:48,473::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000509115 s, 1.5 MB/s\n'; <rc> = 0 >Thread-14::INFO::2014-04-11 22:13:48,476::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-81::ERROR::2014-04-11 22:13:48,484::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-72::DEBUG::2014-04-11 22:13:48,491::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None) >Thread-81::ERROR::2014-04-11 22:13:48,493::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-14::DEBUG::2014-04-11 22:13:48,500::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `f3b51811-4a7f-43af-8633-322b3db23c48` >Thread-82::DEBUG::2014-04-11 22:13:48,513::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-14::INFO::2014-04-11 22:13:48,527::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 21619c8e-99ea-4813-be02-d708971e5393 >Thread-81::DEBUG::2014-04-11 22:13:48,531::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-82::ERROR::2014-04-11 22:13:48,545::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-82::ERROR::2014-04-11 22:13:48,555::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-14::DEBUG::2014-04-11 22:13:48,556::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `21619c8e-99ea-4813-be02-d708971e5393` >Thread-83::DEBUG::2014-04-11 22:13:48,556::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 21619c8e-99ea-4813-be02-d708971e5393 >Thread-74::DEBUG::2014-04-11 22:13:48,558::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None) >Thread-14::DEBUG::2014-04-11 22:13:48,567::misc::772::SamplingMethod::(__call__) Returning last result >Thread-82::DEBUG::2014-04-11 22:13:48,589::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-65::DEBUG::2014-04-11 22:13:48,617::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain a033c2ac-0d01-490c-9552-99ca53d6a64a changed its status to Valid >Thread-88::DEBUG::2014-04-11 22:13:48,661::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-81::DEBUG::2014-04-11 22:13:48,686::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-88::DEBUG::2014-04-11 22:13:48,700::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-88::DEBUG::2014-04-11 22:13:48,745::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-82::DEBUG::2014-04-11 22:13:48,754::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-91::INFO::2014-04-11 22:13:48,775::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-14::DEBUG::2014-04-11 22:13:48,804::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-72::DEBUG::2014-04-11 22:13:48,815::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5 >Thread-14::DEBUG::2014-04-11 22:13:48,852::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-72::WARNING::2014-04-11 22:13:48,866::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8'] >Thread-74::DEBUG::2014-04-11 22:13:48,905::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5 >Thread-73::DEBUG::2014-04-11 22:13:48,941::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-74::WARNING::2014-04-11 22:13:48,955::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce'] >Thread-14::DEBUG::2014-04-11 22:13:49,053::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8 e2ffd939-380e-4fbb-8a23-5a0e69c84a87 b7663d70-e658-41fa-b9f0-8da83c9eddce 3406665e-4adc-4fd4-aa1e-037547b29adb f3b51811-4a7f-43af-8633-322b3db23c48 21619c8e-99ea-4813-be02-d708971e5393' (cwd None) >Thread-81::DEBUG::2014-04-11 22:13:49,102::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-82::DEBUG::2014-04-11 22:13:49,179::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-82::WARNING::2014-04-11 22:13:49,198::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-81::WARNING::2014-04-11 22:13:49,198::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-83::DEBUG::2014-04-11 22:13:49,228::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393 >Thread-73::DEBUG::2014-04-11 22:13:49,407::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-14::DEBUG::2014-04-11 22:13:49,489::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found\n Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87\n Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found\n Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:13:49,492::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8', ' Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found', ' Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87', ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce', ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb', ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48', ' Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found', ' Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393'] >Thread-14::DEBUG::2014-04-11 22:13:49,493::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-73::DEBUG::2014-04-11 22:13:49,590::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_Storage', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=e2ffd939-380e-4fbb-8a23-5a0e69c84a87', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=aa653ed432ff22d61d7b0ce5019a22c0a8bb40dd'] >Thread-73::DEBUG::2014-04-11 22:13:49,621::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-73::DEBUG::2014-04-11 22:13:49,626::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87_imageNS' >Thread-73::DEBUG::2014-04-11 22:13:49,633::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87_volumeNS' >Thread-73::DEBUG::2014-04-11 22:13:49,691::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-73::DEBUG::2014-04-11 22:13:49,710::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000299189 s, 1.2 MB/s\n'; <rc> = 0 >Thread-83::DEBUG::2014-04-11 22:13:49,717::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-73::DEBUG::2014-04-11 22:13:49,725::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 changed its status to Valid >Thread-122::DEBUG::2014-04-11 22:13:49,728::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-122::DEBUG::2014-04-11 22:13:49,729::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-122::DEBUG::2014-04-11 22:13:49,730::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-123::INFO::2014-04-11 22:13:49,732::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-83::DEBUG::2014-04-11 22:13:49,755::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2'] >Thread-83::DEBUG::2014-04-11 22:13:49,760::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-83::DEBUG::2014-04-11 22:13:49,761::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '21619c8e-99ea-4813-be02-d708971e5393_imageNS' >Thread-83::DEBUG::2014-04-11 22:13:49,762::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '21619c8e-99ea-4813-be02-d708971e5393_volumeNS' >Thread-83::DEBUG::2014-04-11 22:13:49,816::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-83::DEBUG::2014-04-11 22:13:49,835::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000273719 s, 2.8 MB/s\n'; <rc> = 0 >Thread-83::DEBUG::2014-04-11 22:13:49,841::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 21619c8e-99ea-4813-be02-d708971e5393 changed its status to Valid >Thread-124::DEBUG::2014-04-11 22:13:49,843::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-124::DEBUG::2014-04-11 22:13:49,843::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-124::DEBUG::2014-04-11 22:13:49,845::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-125::INFO::2014-04-11 22:13:49,846::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-65::DEBUG::2014-04-11 22:13:58,697::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:13:58,720::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000464114 s, 1.6 MB/s\n'; <rc> = 0 >Thread-73::DEBUG::2014-04-11 22:13:59,805::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-73::DEBUG::2014-04-11 22:13:59,823::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000307352 s, 1.2 MB/s\n'; <rc> = 0 >Thread-83::DEBUG::2014-04-11 22:13:59,872::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-83::DEBUG::2014-04-11 22:13:59,892::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00024012 s, 3.1 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:14:08,748::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:14:08,766::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000459543 s, 1.6 MB/s\n'; <rc> = 0 >Thread-73::DEBUG::2014-04-11 22:14:09,880::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-73::DEBUG::2014-04-11 22:14:09,962::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000329968 s, 1.1 MB/s\n'; <rc> = 0 >Thread-83::DEBUG::2014-04-11 22:14:10,005::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-83::DEBUG::2014-04-11 22:14:10,023::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000232803 s, 3.2 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:14:18,794::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:14:18,815::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000355251 s, 2.1 MB/s\n'; <rc> = 0 >Thread-73::DEBUG::2014-04-11 22:14:20,001::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-73::DEBUG::2014-04-11 22:14:20,020::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000234093 s, 1.6 MB/s\n'; <rc> = 0 >Thread-83::DEBUG::2014-04-11 22:14:20,132::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-83::DEBUG::2014-04-11 22:14:20,154::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00024019 s, 3.1 MB/s\n'; <rc> = 0 >Thread-60::WARNING::2014-04-11 22:14:25,948::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-61::WARNING::2014-04-11 22:14:25,952::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-60::DEBUG::2014-04-11 22:14:25,962::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-61::DEBUG::2014-04-11 22:14:25,962::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-65::DEBUG::2014-04-11 22:14:28,844::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:14:28,864::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000589284 s, 1.3 MB/s\n'; <rc> = 0 >Thread-73::DEBUG::2014-04-11 22:14:30,060::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-73::DEBUG::2014-04-11 22:14:30,082::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000281212 s, 1.3 MB/s\n'; <rc> = 0 >Thread-83::DEBUG::2014-04-11 22:14:30,284::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-83::DEBUG::2014-04-11 22:14:30,306::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000212503 s, 3.6 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:14:38,894::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:14:38,914::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000451433 s, 1.7 MB/s\n'; <rc> = 0 >Thread-67::WARNING::2014-04-11 22:14:48,252::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-68::WARNING::2014-04-11 22:14:48,266::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-67::DEBUG::2014-04-11 22:14:48,298::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-68::DEBUG::2014-04-11 22:14:48,333::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-76::WARNING::2014-04-11 22:14:48,466::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-77::WARNING::2014-04-11 22:14:48,475::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-76::DEBUG::2014-04-11 22:14:48,526::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-77::DEBUG::2014-04-11 22:14:48,554::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-85::WARNING::2014-04-11 22:14:48,703::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-86::WARNING::2014-04-11 22:14:48,711::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-85::DEBUG::2014-04-11 22:14:48,757::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-86::DEBUG::2014-04-11 22:14:48,782::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-93::WARNING::2014-04-11 22:14:49,047::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-94::WARNING::2014-04-11 22:14:49,060::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-93::DEBUG::2014-04-11 22:14:49,089::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-94::DEBUG::2014-04-11 22:14:49,108::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-72::ERROR::2014-04-11 22:14:49,136::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-72::ERROR::2014-04-11 22:14:49,160::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-72::DEBUG::2014-04-11 22:14:49,190::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 changed its status to Invalid >Thread-99::WARNING::2014-04-11 22:14:49,185::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-126::DEBUG::2014-04-11 22:14:49,212::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-100::WARNING::2014-04-11 22:14:49,195::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-99::DEBUG::2014-04-11 22:14:49,221::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-126::DEBUG::2014-04-11 22:14:49,230::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-100::DEBUG::2014-04-11 22:14:49,235::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-126::DEBUG::2014-04-11 22:14:49,255::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-74::ERROR::2014-04-11 22:14:49,259::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-74::ERROR::2014-04-11 22:14:49,278::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-74::DEBUG::2014-04-11 22:14:49,301::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain b7663d70-e658-41fa-b9f0-8da83c9eddce changed its status to Invalid >Thread-128::DEBUG::2014-04-11 22:14:49,317::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-128::DEBUG::2014-04-11 22:14:49,331::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-128::DEBUG::2014-04-11 22:14:49,352::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-106::WARNING::2014-04-11 22:14:49,386::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-106::DEBUG::2014-04-11 22:14:49,409::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-107::WARNING::2014-04-11 22:14:49,413::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-107::DEBUG::2014-04-11 22:14:49,436::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-109::WARNING::2014-04-11 22:14:49,438::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-108::WARNING::2014-04-11 22:14:49,443::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-109::DEBUG::2014-04-11 22:14:49,456::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-81::ERROR::2014-04-11 22:14:49,465::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-81::ERROR::2014-04-11 22:14:49,467::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-82::ERROR::2014-04-11 22:14:49,468::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-81::DEBUG::2014-04-11 22:14:49,473::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3406665e-4adc-4fd4-aa1e-037547b29adb changed its status to Invalid >Thread-82::ERROR::2014-04-11 22:14:49,477::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-130::DEBUG::2014-04-11 22:14:49,479::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-82::DEBUG::2014-04-11 22:14:49,482::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain f3b51811-4a7f-43af-8633-322b3db23c48 changed its status to Invalid >Thread-130::DEBUG::2014-04-11 22:14:49,485::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-130::DEBUG::2014-04-11 22:14:49,488::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-65::DEBUG::2014-04-11 22:14:49,489::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-131::DEBUG::2014-04-11 22:14:49,490::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-131::DEBUG::2014-04-11 22:14:49,493::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-131::DEBUG::2014-04-11 22:14:49,509::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-65::DEBUG::2014-04-11 22:14:49,542::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000409051 s, 1.8 MB/s\n'; <rc> = 0 >Thread-117::WARNING::2014-04-11 22:14:49,580::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-117::DEBUG::2014-04-11 22:14:49,585::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-118::WARNING::2014-04-11 22:14:49,588::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-14::INFO::2014-04-11 22:14:49,589::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-14::INFO::2014-04-11 22:14:49,591::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/21619c8e-99ea-4813-be02-d708971e5393 >Thread-14::INFO::2014-04-11 22:14:49,591::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::INFO::2014-04-11 22:14:49,592::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/mastersd >Thread-14::INFO::2014-04-11 22:14:49,593::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:14:49,597::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released >Thread-14::INFO::2014-04-11 22:14:49,597::logUtils::47::dispatcher::(wrapper) Run and protect: reconstructMaster, Return response: None >Thread-14::DEBUG::2014-04-11 22:14:49,597::task::1185::TaskManager.Task::(prepare) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::finished: None >Thread-14::DEBUG::2014-04-11 22:14:49,598::task::595::TaskManager.Task::(_updateState) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:14:49,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>} >Thread-14::DEBUG::2014-04-11 22:14:49,598::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:14:49,599::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:14:49,599::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:14:49,600::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:14:49,600::resourceManager::654::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' has 1 requests in queue. Handling top request. >Thread-14::DEBUG::2014-04-11 22:14:49,600::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`4f06bab6-01cb-4fbf-8307-932a9ee6318c`::Granted request >Thread-14::DEBUG::2014-04-11 22:14:49,601::resourceManager::686::ResourceManager::(releaseResource) Request 'Request for Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c - exclusive: granted' was granted >Thread-28::INFO::2014-04-11 22:14:49,602::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7f4e9c125f38 >Thread-14::DEBUG::2014-04-11 22:14:49,602::task::990::TaskManager.Task::(_decref) Task=`1aea0d71-22f8-4047-9309-48693fc16337`::ref 0 aborting False >Thread-28::INFO::2014-04-11 22:14:49,603::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 38) >Thread-28::DEBUG::2014-04-11 22:14:49,604::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,605::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,605::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,605::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,606::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,606::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,606::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-28::DEBUG::2014-04-11 22:14:49,607::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-28::DEBUG::2014-04-11 22:14:49,607::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-28::DEBUG::2014-04-11 22:14:49,607::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-28::DEBUG::2014-04-11 22:14:49,607::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-28::DEBUG::2014-04-11 22:14:49,608::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-28::DEBUG::2014-04-11 22:14:49,655::misc::772::SamplingMethod::(__call__) Returning last result >Thread-28::DEBUG::2014-04-11 22:14:49,656::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:14:49,698::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [5f90a775] >Thread-14::DEBUG::2014-04-11 22:14:49,699::task::595::TaskManager.Task::(_updateState) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:14:49,699::logUtils::44::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=38, options=None) >Thread-14::DEBUG::2014-04-11 22:14:49,700::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`79d794de-c4a3-46a0-9419-18712d5debb9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '860' at 'refreshStoragePool' >Thread-14::DEBUG::2014-04-11 22:14:49,701::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'shared' >Thread-14::DEBUG::2014-04-11 22:14:49,701::resourceManager::577::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is currently locked, Entering queue (1 in queue) >Thread-28::DEBUG::2014-04-11 22:14:49,785::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-28::DEBUG::2014-04-11 22:14:49,786::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,787::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,787::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,788::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,788::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,789::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,789::misc::772::SamplingMethod::(__call__) Returning last result >Thread-28::ERROR::2014-04-11 22:14:49,789::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-28::ERROR::2014-04-11 22:14:49,790::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-28::DEBUG::2014-04-11 22:14:49,790::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:49,793::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-28::DEBUG::2014-04-11 22:14:50,134::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-28::WARNING::2014-04-11 22:14:50,136::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-28::DEBUG::2014-04-11 22:14:50,137::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:50,198::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-28::DEBUG::2014-04-11 22:14:50,201::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-28::DEBUG::2014-04-11 22:14:50,218::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-28::DEBUG::2014-04-11 22:14:50,221::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-28::INFO::2014-04-11 22:14:50,221::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered >Thread-28::INFO::2014-04-11 22:14:50,221::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered >Thread-28::DEBUG::2014-04-11 22:14:50,222::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38 >Thread-28::DEBUG::2014-04-11 22:14:50,222::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-28::DEBUG::2014-04-11 22:14:50,223::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-28::DEBUG::2014-04-11 22:14:50,238::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-28::DEBUG::2014-04-11 22:14:50,239::misc::772::SamplingMethod::(__call__) Returning last result >Thread-28::WARNING::2014-04-11 22:14:50,240::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c already exists >Thread-28::DEBUG::2014-04-11 22:14:50,256::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-28::DEBUG::2014-04-11 22:14:50,257::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-28::DEBUG::2014-04-11 22:14:50,258::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8 e2ffd939-380e-4fbb-8a23-5a0e69c84a87 b7663d70-e658-41fa-b9f0-8da83c9eddce 3406665e-4adc-4fd4-aa1e-037547b29adb f3b51811-4a7f-43af-8633-322b3db23c48 21619c8e-99ea-4813-be02-d708971e5393' (cwd None) >Thread-28::DEBUG::2014-04-11 22:14:50,557::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found\n Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87\n Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found\n Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393\n'; <rc> = 5 >Thread-28::WARNING::2014-04-11 22:14:50,559::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8', ' Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found', ' Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87', ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce', ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb', ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48', ' Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found', ' Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393'] >Thread-28::DEBUG::2014-04-11 22:14:50,559::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-72::ERROR::2014-04-11 22:14:59,213::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-72::ERROR::2014-04-11 22:14:59,214::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-72::DEBUG::2014-04-11 22:14:59,215::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-72::DEBUG::2014-04-11 22:14:59,216::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None) >Thread-74::ERROR::2014-04-11 22:14:59,319::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-74::ERROR::2014-04-11 22:14:59,320::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-74::DEBUG::2014-04-11 22:14:59,320::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-74::DEBUG::2014-04-11 22:14:59,321::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None) >Thread-81::ERROR::2014-04-11 22:14:59,480::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-81::ERROR::2014-04-11 22:14:59,480::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-81::DEBUG::2014-04-11 22:14:59,481::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-81::DEBUG::2014-04-11 22:14:59,482::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-82::ERROR::2014-04-11 22:14:59,494::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-82::ERROR::2014-04-11 22:14:59,495::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-82::DEBUG::2014-04-11 22:14:59,496::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-82::DEBUG::2014-04-11 22:14:59,498::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-72::DEBUG::2014-04-11 22:14:59,524::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5 >Thread-72::WARNING::2014-04-11 22:14:59,526::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8'] >Thread-74::DEBUG::2014-04-11 22:14:59,647::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5 >Thread-74::WARNING::2014-04-11 22:14:59,660::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce'] >Thread-81::DEBUG::2014-04-11 22:14:59,826::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-82::DEBUG::2014-04-11 22:14:59,834::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-82::WARNING::2014-04-11 22:14:59,845::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-81::WARNING::2014-04-11 22:14:59,846::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-81::DEBUG::2014-04-11 22:14:59,863::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:00,217::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:15:00,305::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000484123 s, 1.6 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:15:10,341::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:15:10,362::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000456852 s, 1.7 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:15:20,392::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:15:20,413::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000591778 s, 1.3 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:15:30,443::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:15:30,469::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000451917 s, 1.7 MB/s\n'; <rc> = 0 >Thread-73::ERROR::2014-04-11 22:15:40,113::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-73::DEBUG::2014-04-11 22:15:40,119::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 changed its status to Invalid >Thread-170::DEBUG::2014-04-11 22:15:40,130::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-73::DEBUG::2014-04-11 22:15:40,133::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-170::DEBUG::2014-04-11 22:15:40,140::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-170::DEBUG::2014-04-11 22:15:40,151::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-83::ERROR::2014-04-11 22:15:40,332::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 21619c8e-99ea-4813-be02-d708971e5393 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 552, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-83::DEBUG::2014-04-11 22:15:40,334::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 21619c8e-99ea-4813-be02-d708971e5393 changed its status to Invalid >Thread-172::DEBUG::2014-04-11 22:15:40,336::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-83::DEBUG::2014-04-11 22:15:40,338::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-172::DEBUG::2014-04-11 22:15:40,342::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-172::DEBUG::2014-04-11 22:15:40,345::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-65::DEBUG::2014-04-11 22:15:40,530::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:15:40,558::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000537818 s, 1.4 MB/s\n'; <rc> = 0 >Thread-135::WARNING::2014-04-11 22:15:50,186::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-137::WARNING::2014-04-11 22:15:50,189::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-138::WARNING::2014-04-11 22:15:50,220::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-136::WARNING::2014-04-11 22:15:50,241::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-141::WARNING::2014-04-11 22:15:50,612::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-142::WARNING::2014-04-11 22:15:50,619::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-143::WARNING::2014-04-11 22:15:50,619::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-144::WARNING::2014-04-11 22:15:50,620::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-28::INFO::2014-04-11 22:15:50,635::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-28::INFO::2014-04-11 22:15:50,640::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/mastersd >Thread-65::DEBUG::2014-04-11 22:15:50,643::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-28::DEBUG::2014-04-11 22:15:50,644::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-28::DEBUG::2014-04-11 22:15:50,671::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-28::DEBUG::2014-04-11 22:15:50,674::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-28::DEBUG::2014-04-11 22:15:50,680::resourceManager::654::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' has 1 requests in queue. Handling top request. >Thread-28::DEBUG::2014-04-11 22:15:50,685::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`79d794de-c4a3-46a0-9419-18712d5debb9`::Granted request >Thread-28::DEBUG::2014-04-11 22:15:50,698::resourceManager::686::ResourceManager::(releaseResource) Request 'Request for Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c - shared: granted' was granted >Thread-28::DEBUG::2014-04-11 22:15:50,707::resourceManager::694::ResourceManager::(releaseResource) This is a shared lock. Granting all shared requests >Thread-14::DEBUG::2014-04-11 22:15:50,718::task::827::TaskManager.Task::(resourceAcquired) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (shared) >Thread-28::DEBUG::2014-04-11 22:15:50,726::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 22:15:50,732::task::990::TaskManager.Task::(_decref) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::ref 1 aborting False >Thread-65::DEBUG::2014-04-11 22:15:50,739::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000395287 s, 1.9 MB/s\n'; <rc> = 0 >Thread-28::DEBUG::2014-04-11 22:15:50,740::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::INFO::2014-04-11 22:15:50,745::fileSD::316::Storage.StorageDomain::(validate) sdUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-28::DEBUG::2014-04-11 22:15:50,753::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-28::DEBUG::2014-04-11 22:15:50,766::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-28::INFO::2014-04-11 22:15:50,769::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-28::DEBUG::2014-04-11 22:15:50,770::task::1185::TaskManager.Task::(prepare) Task=`bb2d61f2-9971-4bcc-9374-5e2746c7e69f`::finished: True >Thread-28::DEBUG::2014-04-11 22:15:50,771::task::595::TaskManager.Task::(_updateState) Task=`bb2d61f2-9971-4bcc-9374-5e2746c7e69f`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:15:50,771::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:15:50,772::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:15:50,773::task::990::TaskManager.Task::(_decref) Task=`bb2d61f2-9971-4bcc-9374-5e2746c7e69f`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:15:50,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-14::DEBUG::2014-04-11 22:15:50,815::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:15:50,832::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:15:50,843::task::595::TaskManager.Task::(_updateState) Task=`daa250f8-7b72-499a-aecb-64852eb2cc9b`::moving from state init -> state preparing >Thread-14::DEBUG::2014-04-11 22:15:50,844::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-28::INFO::2014-04-11 22:15:50,849::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::DEBUG::2014-04-11 22:15:50,855::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:15:50,862::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-28::INFO::2014-04-11 22:15:50,865::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.7', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.7', 'valid': False}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.6', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.4', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000537818', 'lastCheck': '10.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.5', 'valid': False}} >Thread-28::DEBUG::2014-04-11 22:15:50,870::task::1185::TaskManager.Task::(prepare) Task=`daa250f8-7b72-499a-aecb-64852eb2cc9b`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.7', 'valid': False}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.7', 'valid': False}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.6', 'valid': False}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.4', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000537818', 'lastCheck': '10.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.5', 'valid': False}} >Thread-14::DEBUG::2014-04-11 22:15:50,870::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:15:50,875::task::595::TaskManager.Task::(_updateState) Task=`daa250f8-7b72-499a-aecb-64852eb2cc9b`::moving from state preparing -> state finished >Thread-65::DEBUG::2014-04-11 22:15:50,879::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-28::DEBUG::2014-04-11 22:15:50,882::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-65::DEBUG::2014-04-11 22:15:50,885::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-28::DEBUG::2014-04-11 22:15:50,889::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-65::DEBUG::2014-04-11 22:15:50,891::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-65::DEBUG::2014-04-11 22:15:50,897::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-28::DEBUG::2014-04-11 22:15:50,897::task::990::TaskManager.Task::(_decref) Task=`daa250f8-7b72-499a-aecb-64852eb2cc9b`::ref 0 aborting False >Thread-65::DEBUG::2014-04-11 22:15:50,903::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-65::DEBUG::2014-04-11 22:15:50,916::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-65::DEBUG::2014-04-11 22:15:50,968::misc::772::SamplingMethod::(__call__) Returning last result >Thread-65::DEBUG::2014-04-11 22:15:50,973::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-28::DEBUG::2014-04-11 22:15:51,003::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:15:51,011::task::595::TaskManager.Task::(_updateState) Task=`c033a710-6744-4135-886d-243cb5599a54`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:15:51,012::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-65::DEBUG::2014-04-11 22:15:51,155::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:15:51,156::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:51,157::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:51,158::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:51,160::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:51,161::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:51,164::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:51,164::misc::772::SamplingMethod::(__call__) Returning last result >Thread-65::ERROR::2014-04-11 22:15:51,165::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-65::ERROR::2014-04-11 22:15:51,165::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-65::DEBUG::2014-04-11 22:15:51,166::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:51,174::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >Thread-65::DEBUG::2014-04-11 22:15:51,519::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >Thread-65::WARNING::2014-04-11 22:15:51,525::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >Thread-65::DEBUG::2014-04-11 22:15:51,527::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-65::DEBUG::2014-04-11 22:15:51,706::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-65::DEBUG::2014-04-11 22:15:51,711::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-65::DEBUG::2014-04-11 22:15:51,734::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-65::DEBUG::2014-04-11 22:15:51,741::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-65::INFO::2014-04-11 22:15:51,743::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered >Thread-65::INFO::2014-04-11 22:15:51,744::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered >Thread-14::DEBUG::2014-04-11 22:15:51,746::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38 >Thread-14::DEBUG::2014-04-11 22:15:51,753::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-14::DEBUG::2014-04-11 22:15:51,758::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-28::INFO::2014-04-11 22:15:51,765::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-28::DEBUG::2014-04-11 22:15:51,770::task::1185::TaskManager.Task::(prepare) Task=`c033a710-6744-4135-886d-243cb5599a54`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-28::DEBUG::2014-04-11 22:15:51,771::task::595::TaskManager.Task::(_updateState) Task=`c033a710-6744-4135-886d-243cb5599a54`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:15:51,771::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:15:51,772::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:15:51,774::task::990::TaskManager.Task::(_decref) Task=`c033a710-6744-4135-886d-243cb5599a54`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:15:51,793::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-14::DEBUG::2014-04-11 22:15:51,799::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::WARNING::2014-04-11 22:15:51,800::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c already exists >Thread-14::DEBUG::2014-04-11 22:15:51,822::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-14::DEBUG::2014-04-11 22:15:51,836::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:15:51,911::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8 e2ffd939-380e-4fbb-8a23-5a0e69c84a87 b7663d70-e658-41fa-b9f0-8da83c9eddce 3406665e-4adc-4fd4-aa1e-037547b29adb f3b51811-4a7f-43af-8633-322b3db23c48 21619c8e-99ea-4813-be02-d708971e5393' (cwd None) >Thread-28::DEBUG::2014-04-11 22:15:51,946::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:15:51,966::task::595::TaskManager.Task::(_updateState) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:15:51,992::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None) >Thread-28::DEBUG::2014-04-11 22:15:52,013::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d33fe88e-3805-4d7d-835a-1b29b806c56a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart' >Thread-28::DEBUG::2014-04-11 22:15:52,021::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-28::DEBUG::2014-04-11 22:15:52,030::resourceManager::577::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is currently locked, Entering queue (1 in queue) >Thread-14::DEBUG::2014-04-11 22:15:52,225::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found\n Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87\n Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found\n Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393\n'; <rc> = 5 >Thread-14::WARNING::2014-04-11 22:15:52,233::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8', ' Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found', ' Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87', ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce', ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb', ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48', ' Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found', ' Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393'] >Thread-14::DEBUG::2014-04-11 22:15:52,234::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-147::WARNING::2014-04-11 22:15:59,611::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-148::WARNING::2014-04-11 22:15:59,646::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-150::WARNING::2014-04-11 22:15:59,650::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-149::WARNING::2014-04-11 22:15:59,651::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-150::DEBUG::2014-04-11 22:15:59,744::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-149::DEBUG::2014-04-11 22:15:59,767::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-153::WARNING::2014-04-11 22:15:59,779::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-72::ERROR::2014-04-11 22:15:59,814::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-154::WARNING::2014-04-11 22:15:59,812::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-153::DEBUG::2014-04-11 22:15:59,836::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-72::ERROR::2014-04-11 22:15:59,848::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-155::WARNING::2014-04-11 22:15:59,820::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-154::DEBUG::2014-04-11 22:15:59,862::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-156::WARNING::2014-04-11 22:15:59,822::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-155::DEBUG::2014-04-11 22:15:59,900::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-156::DEBUG::2014-04-11 22:15:59,912::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-74::ERROR::2014-04-11 22:15:59,941::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-74::ERROR::2014-04-11 22:15:59,967::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-159::WARNING::2014-04-11 22:15:59,982::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-159::DEBUG::2014-04-11 22:16:00,018::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-161::DEBUG::2014-04-11 22:16:00,025::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-160::WARNING::2014-04-11 22:16:00,015::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-160::DEBUG::2014-04-11 22:16:00,049::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-161::WARNING::2014-04-11 22:16:00,041::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-161::DEBUG::2014-04-11 22:16:00,067::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-162::WARNING::2014-04-11 22:16:00,081::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-162::DEBUG::2014-04-11 22:16:00,101::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-82::ERROR::2014-04-11 22:16:00,115::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-82::ERROR::2014-04-11 22:16:00,126::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-165::WARNING::2014-04-11 22:16:00,145::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-167::WARNING::2014-04-11 22:16:00,147::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-165::DEBUG::2014-04-11 22:16:00,160::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-167::DEBUG::2014-04-11 22:16:00,165::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-166::WARNING::2014-04-11 22:16:00,162::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-168::WARNING::2014-04-11 22:16:00,169::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-166::DEBUG::2014-04-11 22:16:00,174::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-168::DEBUG::2014-04-11 22:16:00,175::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-81::ERROR::2014-04-11 22:16:00,179::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-81::ERROR::2014-04-11 22:16:00,179::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-65::DEBUG::2014-04-11 22:16:01,788::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:16:01,810::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000445846 s, 1.7 MB/s\n'; <rc> = 0 >Thread-72::ERROR::2014-04-11 22:16:09,890::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-72::ERROR::2014-04-11 22:16:09,891::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-72::DEBUG::2014-04-11 22:16:09,891::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-72::DEBUG::2014-04-11 22:16:09,893::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None) >Thread-74::ERROR::2014-04-11 22:16:10,001::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-74::ERROR::2014-04-11 22:16:10,001::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-74::DEBUG::2014-04-11 22:16:10,002::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-74::DEBUG::2014-04-11 22:16:10,003::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None) >Thread-82::ERROR::2014-04-11 22:16:10,142::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-82::ERROR::2014-04-11 22:16:10,142::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-82::DEBUG::2014-04-11 22:16:10,143::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-82::DEBUG::2014-04-11 22:16:10,144::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-81::ERROR::2014-04-11 22:16:10,180::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-81::ERROR::2014-04-11 22:16:10,181::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-81::DEBUG::2014-04-11 22:16:10,182::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-81::DEBUG::2014-04-11 22:16:10,183::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-72::DEBUG::2014-04-11 22:16:10,234::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5 >Thread-72::WARNING::2014-04-11 22:16:10,238::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8'] >Thread-74::DEBUG::2014-04-11 22:16:10,365::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5 >Thread-74::WARNING::2014-04-11 22:16:10,367::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce'] >Thread-82::DEBUG::2014-04-11 22:16:10,504::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-81::DEBUG::2014-04-11 22:16:10,505::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-81::WARNING::2014-04-11 22:16:10,511::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-82::WARNING::2014-04-11 22:16:10,511::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-82::DEBUG::2014-04-11 22:16:10,516::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-65::DEBUG::2014-04-11 22:16:11,840::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:16:11,864::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000356065 s, 2.1 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:16:21,895::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:16:21,918::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.00042857 s, 1.8 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:16:31,949::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:16:31,978::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000454263 s, 1.7 MB/s\n'; <rc> = 0 >Thread-65::DEBUG::2014-04-11 22:16:42,012::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:16:42,042::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000421787 s, 1.8 MB/s\n'; <rc> = 0 >Thread-176::WARNING::2014-04-11 22:16:50,241::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-175::DEBUG::2014-04-11 22:16:50,291::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-177::DEBUG::2014-04-11 22:16:50,292::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-178::WARNING::2014-04-11 22:16:50,283::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-176::DEBUG::2014-04-11 22:16:50,336::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-178::DEBUG::2014-04-11 22:16:50,362::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-175::WARNING::2014-04-11 22:16:50,342::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-177::WARNING::2014-04-11 22:16:50,351::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-175::DEBUG::2014-04-11 22:16:50,417::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-177::DEBUG::2014-04-11 22:16:50,449::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-183::WARNING::2014-04-11 22:16:50,439::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-73::ERROR::2014-04-11 22:16:50,488::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',) >Thread-182::WARNING::2014-04-11 22:16:50,441::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-183::DEBUG::2014-04-11 22:16:50,498::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-181::WARNING::2014-04-11 22:16:50,463::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-73::ERROR::2014-04-11 22:16:50,520::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',) >Thread-184::WARNING::2014-04-11 22:16:50,468::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-182::DEBUG::2014-04-11 22:16:50,531::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-181::DEBUG::2014-04-11 22:16:50,547::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-184::DEBUG::2014-04-11 22:16:50,575::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-83::ERROR::2014-04-11 22:16:50,602::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 21619c8e-99ea-4813-be02-d708971e5393 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',) >Thread-83::ERROR::2014-04-11 22:16:50,623::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 21619c8e-99ea-4813-be02-d708971e5393 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',) >Thread-187::WARNING::2014-04-11 22:16:51,642::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-187::DEBUG::2014-04-11 22:16:51,674::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-188::WARNING::2014-04-11 22:16:51,674::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-189::WARNING::2014-04-11 22:16:51,681::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-188::DEBUG::2014-04-11 22:16:51,710::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-190::WARNING::2014-04-11 22:16:51,686::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-189::DEBUG::2014-04-11 22:16:51,722::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-190::DEBUG::2014-04-11 22:16:51,738::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-193::WARNING::2014-04-11 22:16:52,346::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-193::DEBUG::2014-04-11 22:16:52,374::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-194::WARNING::2014-04-11 22:16:52,383::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-195::WARNING::2014-04-11 22:16:52,387::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-194::DEBUG::2014-04-11 22:16:52,405::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-196::WARNING::2014-04-11 22:16:52,392::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-195::DEBUG::2014-04-11 22:16:52,416::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-196::DEBUG::2014-04-11 22:16:52,426::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-14::INFO::2014-04-11 22:16:52,443::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::INFO::2014-04-11 22:16:52,450::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/mastersd >Thread-14::INFO::2014-04-11 22:16:52,454::logUtils::47::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None >Thread-14::DEBUG::2014-04-11 22:16:52,457::task::1185::TaskManager.Task::(prepare) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::finished: None >Thread-14::DEBUG::2014-04-11 22:16:52,462::task::595::TaskManager.Task::(_updateState) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:16:52,465::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>} >Thread-14::DEBUG::2014-04-11 22:16:52,469::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:16:52,473::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:16:52,480::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:16:52,483::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:16:52,485::resourceManager::654::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' has 1 requests in queue. Handling top request. >Thread-14::DEBUG::2014-04-11 22:16:52,489::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d33fe88e-3805-4d7d-835a-1b29b806c56a`::Granted request >Thread-14::DEBUG::2014-04-11 22:16:52,493::resourceManager::686::ResourceManager::(releaseResource) Request 'Request for Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c - exclusive: granted' was granted >Thread-28::DEBUG::2014-04-11 22:16:52,502::task::827::TaskManager.Task::(resourceAcquired) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive) >Thread-14::DEBUG::2014-04-11 22:16:52,502::task::990::TaskManager.Task::(_decref) Task=`df6759a4-b79e-4eda-aca5-13d5d9d54768`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:16:52,508::task::990::TaskManager.Task::(_decref) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::ref 1 aborting False >Thread-28::INFO::2014-04-11 22:16:52,515::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-28::DEBUG::2014-04-11 22:16:52,520::task::1180::TaskManager.Task::(prepare) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::Prepare: 1 jobs exist, move to acquiring >Thread-28::DEBUG::2014-04-11 22:16:52,522::task::595::TaskManager.Task::(_updateState) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::moving from state preparing -> state acquiring >Thread-28::DEBUG::2014-04-11 22:16:52,527::task::595::TaskManager.Task::(_updateState) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::moving from state acquiring -> state queued >Thread-28::DEBUG::2014-04-11 22:16:52,531::taskManager::50::TaskManager::(_queueTask) queuing task: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:16:52,537::taskManager::56::TaskManager::(_queueTask) task queued: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-6::DEBUG::2014-04-11 22:16:52,538::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-28::DEBUG::2014-04-11 22:16:52,542::task::1182::TaskManager.Task::(prepare) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::returning >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,543::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 running: <bound method Task.commit of <storage.task.Task instance at 0x7f4e840830e0>> with: None >Thread-28::DEBUG::2014-04-11 22:16:52,548::task::990::TaskManager.Task::(_decref) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::ref 0 aborting False >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,549::task::1194::TaskManager.Task::(commit) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::committing task: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,556::task::595::TaskManager.Task::(_updateState) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::moving from state queued -> state running >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,562::task::905::TaskManager.Task::(_runJobs) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f4e6802c6d0>> (args: (-1, '-1', 250, 3) kwargs: {}) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,566::task::331::TaskManager.Task::(run) Task=`9662096e-137a-47a2-8c7f-a26a6b76bcb5`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f4e6802c6d0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,570::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,573::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-65::DEBUG::2014-04-11 22:16:52,586::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,611::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,617::misc::772::SamplingMethod::(__call__) Returning last result >9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:52,639::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,646::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >Thread-65::DEBUG::2014-04-11 22:16:52,655::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n756 bytes (756 B) copied, 0.000528242 s, 1.4 MB/s\n'; <rc> = 0 >9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:52,659::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-28::DEBUG::2014-04-11 22:16:52,675::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [2860bcb8] >Thread-28::DEBUG::2014-04-11 22:16:52,682::task::595::TaskManager.Task::(_updateState) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:16:52,693::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, remove=False, options=None) >Thread-28::ERROR::2014-04-11 22:16:52,700::task::866::TaskManager.Task::(_setError) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 1119, in disconnectStoragePool > self.validateNotSPM(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 320, in validateNotSPM > raise se.IsSpm(spUUID) >IsSpm: Operation not allowed while SPM is active: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) >Thread-28::DEBUG::2014-04-11 22:16:52,713::task::885::TaskManager.Task::(_run) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::Task._run: 6531d861-ec83-4492-8e4b-0673916f1136 ('a106ab81-9d5f-49c1-aeaf-832a137b708c', 1, False) {} failed - stopping task >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,717::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >Thread-28::DEBUG::2014-04-11 22:16:52,721::task::1211::TaskManager.Task::(stop) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::stopping in state preparing (force False) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,726::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully >Thread-28::DEBUG::2014-04-11 22:16:52,732::task::990::TaskManager.Task::(_decref) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::ref 1 aborting True >Thread-28::INFO::2014-04-11 22:16:52,740::task::1168::TaskManager.Task::(prepare) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::aborting: Task is aborted: 'Operation not allowed while SPM is active' - code 656 >Thread-28::DEBUG::2014-04-11 22:16:52,746::task::1173::TaskManager.Task::(prepare) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::Prepare: aborted: Operation not allowed while SPM is active >Thread-28::DEBUG::2014-04-11 22:16:52,751::task::990::TaskManager.Task::(_decref) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::ref 0 aborting True >Thread-28::DEBUG::2014-04-11 22:16:52,755::task::925::TaskManager.Task::(_doAbort) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::Task._doAbort: force False >Thread-28::DEBUG::2014-04-11 22:16:52,760::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:16:52,766::task::595::TaskManager.Task::(_updateState) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::moving from state preparing -> state aborting >Thread-28::DEBUG::2014-04-11 22:16:52,774::task::550::TaskManager.Task::(__state_aborting) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::_aborting: recover policy none >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,774::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=1a9b590b5a7509eee20a56fc1dfccc14eaca7169'] >Thread-28::DEBUG::2014-04-11 22:16:52,781::task::595::TaskManager.Task::(_updateState) Task=`6531d861-ec83-4492-8e4b-0673916f1136`::moving from state aborting -> state failed >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,786::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-28::DEBUG::2014-04-11 22:16:52,787::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:16:52,796::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::ERROR::2014-04-11 22:16:52,802::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Operation not allowed while SPM is active: ('a106ab81-9d5f-49c1-aeaf-832a137b708c',)", 'code': 656}} >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,804::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,816::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,864::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,873::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`7c9c6e9a-712e-4218-a35a-03bc61a13e3a`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool' >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,878::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,884::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,889::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`7c9c6e9a-712e-4218-a35a-03bc61a13e3a`::Granted request >9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:52,898::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a` >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,907::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`1f88c4c9-6777-4dd4-9b40-b052c87a98e0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool' >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,910::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive' >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,917::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,922::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`1f88c4c9-6777-4dd4-9b40-b052c87a98e0`::Granted request >9662096e-137a-47a2-8c7f-a26a6b76bcb5::ERROR::2014-04-11 22:16:52,933::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain a033c2ac-0d01-490c-9552-99ca53d6a64a >9662096e-137a-47a2-8c7f-a26a6b76bcb5::ERROR::2014-04-11 22:16:52,937::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,942::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:52,962::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name a033c2ac-0d01-490c-9552-99ca53d6a64a' (cwd None) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,256::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found\n Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a\n'; <rc> = 5 >9662096e-137a-47a2-8c7f-a26a6b76bcb5::WARNING::2014-04-11 22:16:53,258::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a033c2ac-0d01-490c-9552-99ca53d6a64a" not found', ' Skipping volume group a033c2ac-0d01-490c-9552-99ca53d6a64a'] >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,258::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,394::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,396::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,413::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,415::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:53,416::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered >9662096e-137a-47a2-8c7f-a26a6b76bcb5::INFO::2014-04-11 22:16:53,416::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,417::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,417::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,417::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it. >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,418::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records. >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,418::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,434::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,435::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,435::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,473::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,605::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,662::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-28::DEBUG::2014-04-11 22:16:53,690::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,710::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,734::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms >Thread-28::DEBUG::2014-04-11 22:16:53,735::task::595::TaskManager.Task::(_updateState) Task=`9c288f0f-dd95-46a1-a93a-de30dc2a7166`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:16:53,737::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,737::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks >Thread-28::DEBUG::2014-04-11 22:16:53,737::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:16:53,739::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:16:53,740::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:53,741::task::1185::TaskManager.Task::(prepare) Task=`9c288f0f-dd95-46a1-a93a-de30dc2a7166`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:53,741::task::595::TaskManager.Task::(_updateState) Task=`9c288f0f-dd95-46a1-a93a-de30dc2a7166`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:16:53,742::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:16:53,742::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:16:53,743::task::990::TaskManager.Task::(_decref) Task=`9c288f0f-dd95-46a1-a93a-de30dc2a7166`::ref 0 aborting False >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:16:53,756::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >Thread-28::DEBUG::2014-04-11 22:16:54,749::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:16:54,750::task::595::TaskManager.Task::(_updateState) Task=`32fbd794-660c-49cb-a034-6e7720a8b515`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:16:54,751::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:16:54,751::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:16:54,752::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:16:54,752::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:54,752::task::1185::TaskManager.Task::(prepare) Task=`32fbd794-660c-49cb-a034-6e7720a8b515`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:54,752::task::595::TaskManager.Task::(_updateState) Task=`32fbd794-660c-49cb-a034-6e7720a8b515`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:16:54,753::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:16:54,753::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:16:54,753::task::990::TaskManager.Task::(_decref) Task=`32fbd794-660c-49cb-a034-6e7720a8b515`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:16:55,758::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:16:55,759::task::595::TaskManager.Task::(_updateState) Task=`0e8c79dc-4842-4eef-b3f3-6e41e91a24ff`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:16:55,759::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:16:55,759::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:16:55,760::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:16:55,760::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:55,761::task::1185::TaskManager.Task::(prepare) Task=`0e8c79dc-4842-4eef-b3f3-6e41e91a24ff`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:55,761::task::595::TaskManager.Task::(_updateState) Task=`0e8c79dc-4842-4eef-b3f3-6e41e91a24ff`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:16:55,761::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:16:55,762::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:16:55,762::task::990::TaskManager.Task::(_decref) Task=`0e8c79dc-4842-4eef-b3f3-6e41e91a24ff`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:16:56,767::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:16:56,767::task::595::TaskManager.Task::(_updateState) Task=`80040a28-cb48-46b3-870a-424dd347b05e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:16:56,768::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:16:56,768::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:16:56,769::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:16:56,769::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:56,769::task::1185::TaskManager.Task::(prepare) Task=`80040a28-cb48-46b3-870a-424dd347b05e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:56,769::task::595::TaskManager.Task::(_updateState) Task=`80040a28-cb48-46b3-870a-424dd347b05e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:16:56,770::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:16:56,771::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:16:56,771::task::990::TaskManager.Task::(_decref) Task=`80040a28-cb48-46b3-870a-424dd347b05e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:16:57,775::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:16:57,776::task::595::TaskManager.Task::(_updateState) Task=`69352223-4797-4ac0-a99a-4b3ac120ca30`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:16:57,776::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:16:57,777::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:16:57,777::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:16:57,778::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:57,778::task::1185::TaskManager.Task::(prepare) Task=`69352223-4797-4ac0-a99a-4b3ac120ca30`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:57,778::task::595::TaskManager.Task::(_updateState) Task=`69352223-4797-4ac0-a99a-4b3ac120ca30`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:16:57,778::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:16:57,779::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:16:57,779::task::990::TaskManager.Task::(_decref) Task=`69352223-4797-4ac0-a99a-4b3ac120ca30`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:16:58,784::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:16:58,787::task::595::TaskManager.Task::(_updateState) Task=`9a2cba63-7bf9-4f6f-a7fe-78f6deb42f21`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:16:58,788::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:16:58,789::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:16:58,791::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:16:58,792::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:58,792::task::1185::TaskManager.Task::(prepare) Task=`9a2cba63-7bf9-4f6f-a7fe-78f6deb42f21`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:58,794::task::595::TaskManager.Task::(_updateState) Task=`9a2cba63-7bf9-4f6f-a7fe-78f6deb42f21`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:16:58,794::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:16:58,794::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:16:58,795::task::990::TaskManager.Task::(_decref) Task=`9a2cba63-7bf9-4f6f-a7fe-78f6deb42f21`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:16:59,800::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:16:59,801::task::595::TaskManager.Task::(_updateState) Task=`c1056ca2-752e-4a03-8d6e-1d30dd5226e9`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:16:59,801::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:16:59,801::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:16:59,802::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:16:59,802::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:59,803::task::1185::TaskManager.Task::(prepare) Task=`c1056ca2-752e-4a03-8d6e-1d30dd5226e9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:16:59,803::task::595::TaskManager.Task::(_updateState) Task=`c1056ca2-752e-4a03-8d6e-1d30dd5226e9`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:16:59,803::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:16:59,804::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:16:59,804::task::990::TaskManager.Task::(_decref) Task=`c1056ca2-752e-4a03-8d6e-1d30dd5226e9`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:00,808::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:00,809::task::595::TaskManager.Task::(_updateState) Task=`f4851d03-4282-4a2f-9dd4-4e6f038159dc`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:00,809::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:00,809::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:00,810::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:00,810::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:00,811::task::1185::TaskManager.Task::(prepare) Task=`f4851d03-4282-4a2f-9dd4-4e6f038159dc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:00,811::task::595::TaskManager.Task::(_updateState) Task=`f4851d03-4282-4a2f-9dd4-4e6f038159dc`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:00,811::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:00,812::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:00,812::task::990::TaskManager.Task::(_decref) Task=`f4851d03-4282-4a2f-9dd4-4e6f038159dc`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:01,817::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:01,817::task::595::TaskManager.Task::(_updateState) Task=`9992bda9-0eef-4500-b1fd-f3b7fe31797c`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:01,818::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:01,818::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:01,819::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:01,819::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:01,820::task::1185::TaskManager.Task::(prepare) Task=`9992bda9-0eef-4500-b1fd-f3b7fe31797c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:01,820::task::595::TaskManager.Task::(_updateState) Task=`9992bda9-0eef-4500-b1fd-f3b7fe31797c`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:01,820::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:01,821::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:01,821::task::990::TaskManager.Task::(_decref) Task=`9992bda9-0eef-4500-b1fd-f3b7fe31797c`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:02,826::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:02,839::task::595::TaskManager.Task::(_updateState) Task=`efdc6dc5-510f-4c13-afc6-494ec03becf1`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:02,840::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:02,840::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:02,841::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:02,841::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:02,841::task::1185::TaskManager.Task::(prepare) Task=`efdc6dc5-510f-4c13-afc6-494ec03becf1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:02,842::task::595::TaskManager.Task::(_updateState) Task=`efdc6dc5-510f-4c13-afc6-494ec03becf1`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:02,842::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:02,843::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:02,843::task::990::TaskManager.Task::(_decref) Task=`efdc6dc5-510f-4c13-afc6-494ec03becf1`::ref 0 aborting False >Thread-65::DEBUG::2014-04-11 22:17:02,898::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:17:02,926::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000423116 s, 1.8 MB/s\n'; <rc> = 0 >Thread-28::DEBUG::2014-04-11 22:17:03,851::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:03,859::task::595::TaskManager.Task::(_updateState) Task=`22d3c00a-17b3-4ca2-ac11-874e2523f216`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:03,862::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:03,863::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:03,866::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:03,867::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:03,867::task::1185::TaskManager.Task::(prepare) Task=`22d3c00a-17b3-4ca2-ac11-874e2523f216`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:03,867::task::595::TaskManager.Task::(_updateState) Task=`22d3c00a-17b3-4ca2-ac11-874e2523f216`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:03,872::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:03,873::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:03,874::task::990::TaskManager.Task::(_decref) Task=`22d3c00a-17b3-4ca2-ac11-874e2523f216`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:04,886::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:04,893::task::595::TaskManager.Task::(_updateState) Task=`767f8de3-46ac-4f54-be99-bbc96114b067`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:04,894::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:04,897::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:04,897::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:04,901::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:04,903::task::1185::TaskManager.Task::(prepare) Task=`767f8de3-46ac-4f54-be99-bbc96114b067`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:04,904::task::595::TaskManager.Task::(_updateState) Task=`767f8de3-46ac-4f54-be99-bbc96114b067`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:04,909::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:04,910::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:04,911::task::990::TaskManager.Task::(_decref) Task=`767f8de3-46ac-4f54-be99-bbc96114b067`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:05,921::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:05,922::task::595::TaskManager.Task::(_updateState) Task=`725875c3-5b40-496c-9eec-1c0ef4443a2e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:05,923::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:05,923::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:05,924::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:05,924::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:05,925::task::1185::TaskManager.Task::(prepare) Task=`725875c3-5b40-496c-9eec-1c0ef4443a2e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:05,925::task::595::TaskManager.Task::(_updateState) Task=`725875c3-5b40-496c-9eec-1c0ef4443a2e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:05,926::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:05,926::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:05,927::task::990::TaskManager.Task::(_decref) Task=`725875c3-5b40-496c-9eec-1c0ef4443a2e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:06,931::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:06,933::task::595::TaskManager.Task::(_updateState) Task=`471f94b3-32bd-40ad-8691-37ea9c71609e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:06,933::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:06,934::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:06,934::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:06,935::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:06,936::task::1185::TaskManager.Task::(prepare) Task=`471f94b3-32bd-40ad-8691-37ea9c71609e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:06,937::task::595::TaskManager.Task::(_updateState) Task=`471f94b3-32bd-40ad-8691-37ea9c71609e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:06,937::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:06,937::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:06,938::task::990::TaskManager.Task::(_decref) Task=`471f94b3-32bd-40ad-8691-37ea9c71609e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:07,943::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:07,958::task::595::TaskManager.Task::(_updateState) Task=`c3636159-aa3c-4fe9-b9da-55bfc1e93c9f`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:07,959::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:07,960::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:07,960::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:07,961::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:07,962::task::1185::TaskManager.Task::(prepare) Task=`c3636159-aa3c-4fe9-b9da-55bfc1e93c9f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:07,963::task::595::TaskManager.Task::(_updateState) Task=`c3636159-aa3c-4fe9-b9da-55bfc1e93c9f`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:07,963::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:07,964::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:07,965::task::990::TaskManager.Task::(_decref) Task=`c3636159-aa3c-4fe9-b9da-55bfc1e93c9f`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:08,972::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:08,978::task::595::TaskManager.Task::(_updateState) Task=`a658eef8-ad6d-4951-a696-8151371ba65d`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:08,981::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:08,982::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:08,984::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:08,986::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:08,988::task::1185::TaskManager.Task::(prepare) Task=`a658eef8-ad6d-4951-a696-8151371ba65d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:08,990::task::595::TaskManager.Task::(_updateState) Task=`a658eef8-ad6d-4951-a696-8151371ba65d`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:08,991::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:08,993::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:08,995::task::990::TaskManager.Task::(_decref) Task=`a658eef8-ad6d-4951-a696-8151371ba65d`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:10,016::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:10,031::task::595::TaskManager.Task::(_updateState) Task=`9c0bdede-a9e6-4d87-93ca-f5ac92f24e92`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:10,040::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:10,040::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:10,056::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:10,060::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:10,063::task::1185::TaskManager.Task::(prepare) Task=`9c0bdede-a9e6-4d87-93ca-f5ac92f24e92`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:10,070::task::595::TaskManager.Task::(_updateState) Task=`9c0bdede-a9e6-4d87-93ca-f5ac92f24e92`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:10,074::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:10,077::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:10,084::task::990::TaskManager.Task::(_decref) Task=`9c0bdede-a9e6-4d87-93ca-f5ac92f24e92`::ref 0 aborting False >Thread-199::WARNING::2014-04-11 22:17:10,339::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-200::WARNING::2014-04-11 22:17:10,369::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-199::DEBUG::2014-04-11 22:17:10,413::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-202::WARNING::2014-04-11 22:17:10,374::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-200::DEBUG::2014-04-11 22:17:10,444::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-201::WARNING::2014-04-11 22:17:10,374::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-202::DEBUG::2014-04-11 22:17:10,476::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-205::WARNING::2014-04-11 22:17:10,472::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-201::DEBUG::2014-04-11 22:17:10,506::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-206::WARNING::2014-04-11 22:17:10,495::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-205::DEBUG::2014-04-11 22:17:10,532::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-72::ERROR::2014-04-11 22:17:10,549::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-207::WARNING::2014-04-11 22:17:10,502::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-206::DEBUG::2014-04-11 22:17:10,558::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-208::WARNING::2014-04-11 22:17:10,503::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-72::ERROR::2014-04-11 22:17:10,585::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-228::ERROR::2014-04-11 22:17:10,588::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-207::DEBUG::2014-04-11 22:17:10,598::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-208::DEBUG::2014-04-11 22:17:10,611::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-228::ERROR::2014-04-11 22:17:10,635::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-211::WARNING::2014-04-11 22:17:10,637::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-74::ERROR::2014-04-11 22:17:10,662::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-228::DEBUG::2014-04-11 22:17:10,670::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-211::DEBUG::2014-04-11 22:17:10,683::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-212::WARNING::2014-04-11 22:17:10,668::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-74::ERROR::2014-04-11 22:17:10,695::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-230::ERROR::2014-04-11 22:17:10,696::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-213::WARNING::2014-04-11 22:17:10,679::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-212::DEBUG::2014-04-11 22:17:10,716::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-214::WARNING::2014-04-11 22:17:10,714::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-230::ERROR::2014-04-11 22:17:10,731::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-213::DEBUG::2014-04-11 22:17:10,740::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-214::DEBUG::2014-04-11 22:17:10,751::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-230::DEBUG::2014-04-11 22:17:10,762::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-81::ERROR::2014-04-11 22:17:10,778::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-217::WARNING::2014-04-11 22:17:10,780::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-228::DEBUG::2014-04-11 22:17:10,796::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None) >Thread-81::ERROR::2014-04-11 22:17:10,802::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-231::ERROR::2014-04-11 22:17:10,803::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-217::DEBUG::2014-04-11 22:17:10,834::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-218::WARNING::2014-04-11 22:17:10,807::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-219::WARNING::2014-04-11 22:17:10,809::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-231::ERROR::2014-04-11 22:17:10,850::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-220::WARNING::2014-04-11 22:17:10,811::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-218::DEBUG::2014-04-11 22:17:10,862::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-219::DEBUG::2014-04-11 22:17:10,865::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-230::DEBUG::2014-04-11 22:17:10,872::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None) >Thread-231::DEBUG::2014-04-11 22:17:10,872::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-220::DEBUG::2014-04-11 22:17:10,874::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-82::ERROR::2014-04-11 22:17:10,909::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-82::ERROR::2014-04-11 22:17:10,918::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-232::ERROR::2014-04-11 22:17:10,920::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-232::ERROR::2014-04-11 22:17:10,933::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-231::DEBUG::2014-04-11 22:17:10,940::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-232::DEBUG::2014-04-11 22:17:10,946::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-232::DEBUG::2014-04-11 22:17:10,976::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-28::DEBUG::2014-04-11 22:17:11,112::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:11,113::task::595::TaskManager.Task::(_updateState) Task=`9643246b-0319-44ef-9775-9e99cf35739d`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:11,113::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:11,114::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:11,115::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-228::DEBUG::2014-04-11 22:17:11,116::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5 >Thread-28::INFO::2014-04-11 22:17:11,134::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:11,138::task::1185::TaskManager.Task::(prepare) Task=`9643246b-0319-44ef-9775-9e99cf35739d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:11,140::task::595::TaskManager.Task::(_updateState) Task=`9643246b-0319-44ef-9775-9e99cf35739d`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:11,142::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:11,143::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-228::WARNING::2014-04-11 22:17:11,144::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8'] >Thread-28::DEBUG::2014-04-11 22:17:11,144::task::990::TaskManager.Task::(_decref) Task=`9643246b-0319-44ef-9775-9e99cf35739d`::ref 0 aborting False >Thread-230::DEBUG::2014-04-11 22:17:11,266::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5 >Thread-231::DEBUG::2014-04-11 22:17:11,275::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-232::DEBUG::2014-04-11 22:17:11,275::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-230::WARNING::2014-04-11 22:17:11,280::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce'] >Thread-232::WARNING::2014-04-11 22:17:11,280::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-231::WARNING::2014-04-11 22:17:11,280::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-231::DEBUG::2014-04-11 22:17:11,287::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:17:12,157::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:12,158::task::595::TaskManager.Task::(_updateState) Task=`e15eb6c4-5e0a-4744-b8da-8c87be236f29`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:12,159::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:12,159::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:12,160::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:12,160::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:12,161::task::1185::TaskManager.Task::(prepare) Task=`e15eb6c4-5e0a-4744-b8da-8c87be236f29`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:12,161::task::595::TaskManager.Task::(_updateState) Task=`e15eb6c4-5e0a-4744-b8da-8c87be236f29`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:12,162::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:12,162::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:12,162::task::990::TaskManager.Task::(_decref) Task=`e15eb6c4-5e0a-4744-b8da-8c87be236f29`::ref 0 aborting False >Thread-65::DEBUG::2014-04-11 22:17:12,958::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:17:13,009::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000366744 s, 2.1 MB/s\n'; <rc> = 0 >Thread-28::DEBUG::2014-04-11 22:17:13,171::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:13,175::task::595::TaskManager.Task::(_updateState) Task=`2140cbba-f0c6-4d9e-8252-d8e7dbf2034d`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:13,177::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:13,179::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:13,184::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:13,187::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:13,189::task::1185::TaskManager.Task::(prepare) Task=`2140cbba-f0c6-4d9e-8252-d8e7dbf2034d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:13,192::task::595::TaskManager.Task::(_updateState) Task=`2140cbba-f0c6-4d9e-8252-d8e7dbf2034d`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:13,198::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:13,201::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:13,203::task::990::TaskManager.Task::(_decref) Task=`2140cbba-f0c6-4d9e-8252-d8e7dbf2034d`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:14,214::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:14,216::task::595::TaskManager.Task::(_updateState) Task=`d2cde899-4c6e-4f88-b2c7-01f2db97a78a`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:14,217::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:14,217::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:14,217::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:14,218::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:14,218::task::1185::TaskManager.Task::(prepare) Task=`d2cde899-4c6e-4f88-b2c7-01f2db97a78a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:14,218::task::595::TaskManager.Task::(_updateState) Task=`d2cde899-4c6e-4f88-b2c7-01f2db97a78a`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:14,219::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:14,219::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:14,221::task::990::TaskManager.Task::(_decref) Task=`d2cde899-4c6e-4f88-b2c7-01f2db97a78a`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:15,226::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:15,227::task::595::TaskManager.Task::(_updateState) Task=`aedde964-e4b5-4e2a-bce4-3081f1c2d995`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:15,228::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:15,228::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:15,228::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:15,229::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:15,229::task::1185::TaskManager.Task::(prepare) Task=`aedde964-e4b5-4e2a-bce4-3081f1c2d995`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:15,229::task::595::TaskManager.Task::(_updateState) Task=`aedde964-e4b5-4e2a-bce4-3081f1c2d995`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:15,230::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:15,230::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:15,230::task::990::TaskManager.Task::(_decref) Task=`aedde964-e4b5-4e2a-bce4-3081f1c2d995`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:16,234::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:16,235::task::595::TaskManager.Task::(_updateState) Task=`00a4e8a2-bb96-48a8-8d11-45105df110d6`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:16,235::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:16,235::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:16,236::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:16,236::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:16,237::task::1185::TaskManager.Task::(prepare) Task=`00a4e8a2-bb96-48a8-8d11-45105df110d6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:16,237::task::595::TaskManager.Task::(_updateState) Task=`00a4e8a2-bb96-48a8-8d11-45105df110d6`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:16,237::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:16,238::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:16,238::task::990::TaskManager.Task::(_decref) Task=`00a4e8a2-bb96-48a8-8d11-45105df110d6`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:17,242::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:17,243::task::595::TaskManager.Task::(_updateState) Task=`b2eda07b-3d85-4d8a-aa63-f2609c9f0aef`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:17,243::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:17,244::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:17,244::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:17,244::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:17,245::task::1185::TaskManager.Task::(prepare) Task=`b2eda07b-3d85-4d8a-aa63-f2609c9f0aef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:17,245::task::595::TaskManager.Task::(_updateState) Task=`b2eda07b-3d85-4d8a-aa63-f2609c9f0aef`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:17,246::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:17,246::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:17,246::task::990::TaskManager.Task::(_decref) Task=`b2eda07b-3d85-4d8a-aa63-f2609c9f0aef`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:18,250::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:18,251::task::595::TaskManager.Task::(_updateState) Task=`221f94e3-c676-43e2-9977-ffa70f10549c`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:18,252::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:18,252::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:18,252::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:18,253::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:18,253::task::1185::TaskManager.Task::(prepare) Task=`221f94e3-c676-43e2-9977-ffa70f10549c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:18,253::task::595::TaskManager.Task::(_updateState) Task=`221f94e3-c676-43e2-9977-ffa70f10549c`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:18,254::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:18,254::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:18,254::task::990::TaskManager.Task::(_decref) Task=`221f94e3-c676-43e2-9977-ffa70f10549c`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:19,259::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:19,261::task::595::TaskManager.Task::(_updateState) Task=`696c9964-74ac-4de8-8f2e-2e4434a2da59`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:19,261::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:19,262::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:19,262::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:19,262::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:19,263::task::1185::TaskManager.Task::(prepare) Task=`696c9964-74ac-4de8-8f2e-2e4434a2da59`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:19,264::task::595::TaskManager.Task::(_updateState) Task=`696c9964-74ac-4de8-8f2e-2e4434a2da59`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:19,264::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:19,265::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:19,265::task::990::TaskManager.Task::(_decref) Task=`696c9964-74ac-4de8-8f2e-2e4434a2da59`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:20,269::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:20,270::task::595::TaskManager.Task::(_updateState) Task=`9061c9eb-2c40-4581-bb23-6d23f76f0eca`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:20,270::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:20,271::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:20,271::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:20,271::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:20,272::task::1185::TaskManager.Task::(prepare) Task=`9061c9eb-2c40-4581-bb23-6d23f76f0eca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:20,272::task::595::TaskManager.Task::(_updateState) Task=`9061c9eb-2c40-4581-bb23-6d23f76f0eca`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:20,272::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:20,273::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:20,273::task::990::TaskManager.Task::(_decref) Task=`9061c9eb-2c40-4581-bb23-6d23f76f0eca`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:21,278::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:21,280::task::595::TaskManager.Task::(_updateState) Task=`1bcb6b0f-f6ef-45f2-aaee-4434d5eb7763`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:21,281::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:21,281::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:21,281::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:21,282::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:21,282::task::1185::TaskManager.Task::(prepare) Task=`1bcb6b0f-f6ef-45f2-aaee-4434d5eb7763`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:21,282::task::595::TaskManager.Task::(_updateState) Task=`1bcb6b0f-f6ef-45f2-aaee-4434d5eb7763`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:21,283::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:21,283::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:21,284::task::990::TaskManager.Task::(_decref) Task=`1bcb6b0f-f6ef-45f2-aaee-4434d5eb7763`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:22,288::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:22,288::task::595::TaskManager.Task::(_updateState) Task=`3e02b295-9a2d-4101-831c-0d5c63eb4f41`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:22,289::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:22,289::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:22,289::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:22,290::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:22,290::task::1185::TaskManager.Task::(prepare) Task=`3e02b295-9a2d-4101-831c-0d5c63eb4f41`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:22,290::task::595::TaskManager.Task::(_updateState) Task=`3e02b295-9a2d-4101-831c-0d5c63eb4f41`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:22,291::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:22,291::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:22,291::task::990::TaskManager.Task::(_decref) Task=`3e02b295-9a2d-4101-831c-0d5c63eb4f41`::ref 0 aborting False >Thread-65::DEBUG::2014-04-11 22:17:23,056::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:17:23,079::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000389197 s, 1.9 MB/s\n'; <rc> = 0 >Thread-28::DEBUG::2014-04-11 22:17:23,299::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:23,303::task::595::TaskManager.Task::(_updateState) Task=`de327f1b-eb84-458f-9d22-00e658170711`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:23,306::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:23,307::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:23,309::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:23,311::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:23,314::task::1185::TaskManager.Task::(prepare) Task=`de327f1b-eb84-458f-9d22-00e658170711`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:23,320::task::595::TaskManager.Task::(_updateState) Task=`de327f1b-eb84-458f-9d22-00e658170711`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:23,322::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:23,325::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:23,327::task::990::TaskManager.Task::(_decref) Task=`de327f1b-eb84-458f-9d22-00e658170711`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:24,338::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:24,340::task::595::TaskManager.Task::(_updateState) Task=`674c1648-f821-4d50-a25b-a5f66d4680a2`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:24,340::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:24,341::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:24,341::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:24,341::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:24,342::task::1185::TaskManager.Task::(prepare) Task=`674c1648-f821-4d50-a25b-a5f66d4680a2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:24,342::task::595::TaskManager.Task::(_updateState) Task=`674c1648-f821-4d50-a25b-a5f66d4680a2`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:24,342::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:24,343::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:24,343::task::990::TaskManager.Task::(_decref) Task=`674c1648-f821-4d50-a25b-a5f66d4680a2`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:25,348::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:25,348::task::595::TaskManager.Task::(_updateState) Task=`c6b4f47f-ab4b-477d-a37c-c84055396e01`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:25,349::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:25,349::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:25,349::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:25,350::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:25,350::task::1185::TaskManager.Task::(prepare) Task=`c6b4f47f-ab4b-477d-a37c-c84055396e01`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:25,350::task::595::TaskManager.Task::(_updateState) Task=`c6b4f47f-ab4b-477d-a37c-c84055396e01`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:25,351::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:25,351::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:25,351::task::990::TaskManager.Task::(_decref) Task=`c6b4f47f-ab4b-477d-a37c-c84055396e01`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:26,356::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:26,395::task::595::TaskManager.Task::(_updateState) Task=`ce1afedb-1466-4658-b7ca-aa16c9b057e4`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:26,396::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:26,396::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:26,396::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:26,397::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:26,397::task::1185::TaskManager.Task::(prepare) Task=`ce1afedb-1466-4658-b7ca-aa16c9b057e4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:26,397::task::595::TaskManager.Task::(_updateState) Task=`ce1afedb-1466-4658-b7ca-aa16c9b057e4`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:26,398::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:26,398::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:26,398::task::990::TaskManager.Task::(_decref) Task=`ce1afedb-1466-4658-b7ca-aa16c9b057e4`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:27,403::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:27,404::task::595::TaskManager.Task::(_updateState) Task=`c299b807-2276-477a-8b11-3786dff393c7`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:27,405::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:27,405::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:27,405::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:27,406::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:27,406::task::1185::TaskManager.Task::(prepare) Task=`c299b807-2276-477a-8b11-3786dff393c7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:27,406::task::595::TaskManager.Task::(_updateState) Task=`c299b807-2276-477a-8b11-3786dff393c7`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:27,406::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:27,407::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:27,407::task::990::TaskManager.Task::(_decref) Task=`c299b807-2276-477a-8b11-3786dff393c7`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:28,411::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:28,412::task::595::TaskManager.Task::(_updateState) Task=`d142cf48-6c32-4efe-a48a-e9e4907018be`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:28,413::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:28,413::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:28,413::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:28,414::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:28,414::task::1185::TaskManager.Task::(prepare) Task=`d142cf48-6c32-4efe-a48a-e9e4907018be`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:28,414::task::595::TaskManager.Task::(_updateState) Task=`d142cf48-6c32-4efe-a48a-e9e4907018be`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:28,415::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:28,415::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:28,415::task::990::TaskManager.Task::(_decref) Task=`d142cf48-6c32-4efe-a48a-e9e4907018be`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:29,421::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:29,423::task::595::TaskManager.Task::(_updateState) Task=`5c6cc6cc-2c7f-4370-8e67-2b168b87deab`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:29,424::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:29,425::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:29,425::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:29,426::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:29,427::task::1185::TaskManager.Task::(prepare) Task=`5c6cc6cc-2c7f-4370-8e67-2b168b87deab`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:29,427::task::595::TaskManager.Task::(_updateState) Task=`5c6cc6cc-2c7f-4370-8e67-2b168b87deab`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:29,428::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:29,428::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:29,429::task::990::TaskManager.Task::(_decref) Task=`5c6cc6cc-2c7f-4370-8e67-2b168b87deab`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:30,445::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:30,445::task::595::TaskManager.Task::(_updateState) Task=`0debfd02-50c6-41ae-878d-07c4726f4cbb`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:30,446::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:30,446::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:30,446::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:30,447::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:30,447::task::1185::TaskManager.Task::(prepare) Task=`0debfd02-50c6-41ae-878d-07c4726f4cbb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:30,447::task::595::TaskManager.Task::(_updateState) Task=`0debfd02-50c6-41ae-878d-07c4726f4cbb`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:30,448::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:30,448::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:30,448::task::990::TaskManager.Task::(_decref) Task=`0debfd02-50c6-41ae-878d-07c4726f4cbb`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:31,453::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:31,456::task::595::TaskManager.Task::(_updateState) Task=`04ef7eae-1298-4a0b-a2c5-58bb9682b4fa`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:31,457::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:31,457::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:31,458::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:31,458::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:31,458::task::1185::TaskManager.Task::(prepare) Task=`04ef7eae-1298-4a0b-a2c5-58bb9682b4fa`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:31,459::task::595::TaskManager.Task::(_updateState) Task=`04ef7eae-1298-4a0b-a2c5-58bb9682b4fa`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:31,459::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:31,460::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:31,460::task::990::TaskManager.Task::(_decref) Task=`04ef7eae-1298-4a0b-a2c5-58bb9682b4fa`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:32,464::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:32,465::task::595::TaskManager.Task::(_updateState) Task=`17d3c09b-9f71-407b-bb74-1f701f6f416b`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:32,465::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:32,465::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:32,466::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:32,466::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:32,466::task::1185::TaskManager.Task::(prepare) Task=`17d3c09b-9f71-407b-bb74-1f701f6f416b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:32,467::task::595::TaskManager.Task::(_updateState) Task=`17d3c09b-9f71-407b-bb74-1f701f6f416b`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:32,467::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:32,467::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:32,468::task::990::TaskManager.Task::(_decref) Task=`17d3c09b-9f71-407b-bb74-1f701f6f416b`::ref 0 aborting False >Thread-65::DEBUG::2014-04-11 22:17:33,112::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:17:33,142::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000448966 s, 1.7 MB/s\n'; <rc> = 0 >Thread-28::DEBUG::2014-04-11 22:17:33,472::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:33,473::task::595::TaskManager.Task::(_updateState) Task=`7835a927-a67b-472b-96e0-8e8c4a3d9b3e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:33,474::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:33,475::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:33,476::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:33,476::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:33,477::task::1185::TaskManager.Task::(prepare) Task=`7835a927-a67b-472b-96e0-8e8c4a3d9b3e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:33,477::task::595::TaskManager.Task::(_updateState) Task=`7835a927-a67b-472b-96e0-8e8c4a3d9b3e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:33,478::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:33,478::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:33,478::task::990::TaskManager.Task::(_decref) Task=`7835a927-a67b-472b-96e0-8e8c4a3d9b3e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:34,485::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:34,487::task::595::TaskManager.Task::(_updateState) Task=`3bbe0ca8-277e-4853-bcf9-26051b50bab3`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:34,488::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:34,489::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:34,489::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:34,490::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:34,491::task::1185::TaskManager.Task::(prepare) Task=`3bbe0ca8-277e-4853-bcf9-26051b50bab3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:34,491::task::595::TaskManager.Task::(_updateState) Task=`3bbe0ca8-277e-4853-bcf9-26051b50bab3`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:34,492::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:34,495::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:34,496::task::990::TaskManager.Task::(_decref) Task=`3bbe0ca8-277e-4853-bcf9-26051b50bab3`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:35,501::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:35,503::task::595::TaskManager.Task::(_updateState) Task=`f65d565e-0240-4491-b524-4094c1e7ae9c`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:35,504::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:35,505::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:35,505::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:35,508::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:35,509::task::1185::TaskManager.Task::(prepare) Task=`f65d565e-0240-4491-b524-4094c1e7ae9c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:35,510::task::595::TaskManager.Task::(_updateState) Task=`f65d565e-0240-4491-b524-4094c1e7ae9c`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:35,510::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:35,511::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:35,511::task::990::TaskManager.Task::(_decref) Task=`f65d565e-0240-4491-b524-4094c1e7ae9c`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:36,517::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:36,520::task::595::TaskManager.Task::(_updateState) Task=`29db3d1c-1d78-4250-9073-3aa00b58e071`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:36,520::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:36,521::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:36,521::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:36,521::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:36,522::task::1185::TaskManager.Task::(prepare) Task=`29db3d1c-1d78-4250-9073-3aa00b58e071`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:36,522::task::595::TaskManager.Task::(_updateState) Task=`29db3d1c-1d78-4250-9073-3aa00b58e071`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:36,522::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:36,522::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:36,523::task::990::TaskManager.Task::(_decref) Task=`29db3d1c-1d78-4250-9073-3aa00b58e071`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:37,527::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:37,528::task::595::TaskManager.Task::(_updateState) Task=`2c45668b-7952-40e0-a1d3-5440c3b84212`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:37,528::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:37,528::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:37,529::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:37,529::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:37,529::task::1185::TaskManager.Task::(prepare) Task=`2c45668b-7952-40e0-a1d3-5440c3b84212`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:37,530::task::595::TaskManager.Task::(_updateState) Task=`2c45668b-7952-40e0-a1d3-5440c3b84212`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:37,530::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:37,530::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:37,531::task::990::TaskManager.Task::(_decref) Task=`2c45668b-7952-40e0-a1d3-5440c3b84212`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:38,535::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:38,536::task::595::TaskManager.Task::(_updateState) Task=`0ff7754a-3578-4837-a642-61f61033ca96`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:38,536::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:38,536::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:38,536::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:38,537::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:38,537::task::1185::TaskManager.Task::(prepare) Task=`0ff7754a-3578-4837-a642-61f61033ca96`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:38,538::task::595::TaskManager.Task::(_updateState) Task=`0ff7754a-3578-4837-a642-61f61033ca96`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:38,538::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:38,538::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:38,539::task::990::TaskManager.Task::(_decref) Task=`0ff7754a-3578-4837-a642-61f61033ca96`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:39,545::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:39,547::task::595::TaskManager.Task::(_updateState) Task=`7f1b927a-ff72-4a51-938e-4fc8a7d58a26`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:39,548::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:39,549::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:39,549::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:39,549::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:39,550::task::1185::TaskManager.Task::(prepare) Task=`7f1b927a-ff72-4a51-938e-4fc8a7d58a26`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:39,550::task::595::TaskManager.Task::(_updateState) Task=`7f1b927a-ff72-4a51-938e-4fc8a7d58a26`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:39,551::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:39,551::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:39,552::task::990::TaskManager.Task::(_decref) Task=`7f1b927a-ff72-4a51-938e-4fc8a7d58a26`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:40,557::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:40,559::task::595::TaskManager.Task::(_updateState) Task=`f9930b68-a94f-4385-a1ed-054a982a990a`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:40,559::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:40,560::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:40,561::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:40,561::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:40,561::task::1185::TaskManager.Task::(prepare) Task=`f9930b68-a94f-4385-a1ed-054a982a990a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:40,563::task::595::TaskManager.Task::(_updateState) Task=`f9930b68-a94f-4385-a1ed-054a982a990a`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:40,564::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:40,564::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:40,564::task::990::TaskManager.Task::(_decref) Task=`f9930b68-a94f-4385-a1ed-054a982a990a`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:41,570::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:41,588::task::595::TaskManager.Task::(_updateState) Task=`d7a78e22-90c8-4c81-9e98-2392f1b5d495`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:41,589::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:41,590::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:41,590::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:41,590::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:41,591::task::1185::TaskManager.Task::(prepare) Task=`d7a78e22-90c8-4c81-9e98-2392f1b5d495`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:41,592::task::595::TaskManager.Task::(_updateState) Task=`d7a78e22-90c8-4c81-9e98-2392f1b5d495`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:41,592::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:41,592::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:41,593::task::990::TaskManager.Task::(_decref) Task=`d7a78e22-90c8-4c81-9e98-2392f1b5d495`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:42,598::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:42,599::task::595::TaskManager.Task::(_updateState) Task=`e119d0a6-392e-44c2-a49e-3794a86acb13`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:42,599::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:42,599::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:42,600::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:42,600::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:42,601::task::1185::TaskManager.Task::(prepare) Task=`e119d0a6-392e-44c2-a49e-3794a86acb13`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:42,601::task::595::TaskManager.Task::(_updateState) Task=`e119d0a6-392e-44c2-a49e-3794a86acb13`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:42,602::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:42,602::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:42,603::task::990::TaskManager.Task::(_decref) Task=`e119d0a6-392e-44c2-a49e-3794a86acb13`::ref 0 aborting False >Thread-65::DEBUG::2014-04-11 22:17:43,175::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-65::DEBUG::2014-04-11 22:17:43,205::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000400997 s, 1.9 MB/s\n'; <rc> = 0 >Thread-28::DEBUG::2014-04-11 22:17:43,609::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:43,610::task::595::TaskManager.Task::(_updateState) Task=`71d18ecc-9964-4b89-b280-d5572d73efb8`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:43,611::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:43,611::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:43,612::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:43,612::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:43,612::task::1185::TaskManager.Task::(prepare) Task=`71d18ecc-9964-4b89-b280-d5572d73efb8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:43,614::task::595::TaskManager.Task::(_updateState) Task=`71d18ecc-9964-4b89-b280-d5572d73efb8`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:43,616::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:43,617::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:43,619::task::990::TaskManager.Task::(_decref) Task=`71d18ecc-9964-4b89-b280-d5572d73efb8`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:44,629::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:44,631::task::595::TaskManager.Task::(_updateState) Task=`42e7bbb5-9579-4976-a718-9014192c2ac0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:44,632::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:44,633::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:44,634::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:44,634::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:44,637::task::1185::TaskManager.Task::(prepare) Task=`42e7bbb5-9579-4976-a718-9014192c2ac0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:44,639::task::595::TaskManager.Task::(_updateState) Task=`42e7bbb5-9579-4976-a718-9014192c2ac0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:44,640::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:44,641::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:44,642::task::990::TaskManager.Task::(_decref) Task=`42e7bbb5-9579-4976-a718-9014192c2ac0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:45,652::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:45,653::task::595::TaskManager.Task::(_updateState) Task=`95014451-74ca-492c-8a63-895998fbcb9b`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:45,654::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:45,654::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:45,654::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:45,655::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:45,655::task::1185::TaskManager.Task::(prepare) Task=`95014451-74ca-492c-8a63-895998fbcb9b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:45,656::task::595::TaskManager.Task::(_updateState) Task=`95014451-74ca-492c-8a63-895998fbcb9b`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:45,656::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:45,656::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:45,657::task::990::TaskManager.Task::(_decref) Task=`95014451-74ca-492c-8a63-895998fbcb9b`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:46,661::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:46,680::task::595::TaskManager.Task::(_updateState) Task=`664a10a5-edf1-40c2-9cea-372a1ba71b27`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:46,681::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:46,681::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:46,682::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:46,682::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:46,682::task::1185::TaskManager.Task::(prepare) Task=`664a10a5-edf1-40c2-9cea-372a1ba71b27`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:46,683::task::595::TaskManager.Task::(_updateState) Task=`664a10a5-edf1-40c2-9cea-372a1ba71b27`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:46,683::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:46,684::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:46,684::task::990::TaskManager.Task::(_decref) Task=`664a10a5-edf1-40c2-9cea-372a1ba71b27`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:47,689::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:47,690::task::595::TaskManager.Task::(_updateState) Task=`450b0a22-f873-41c2-90b4-a908efdc52fd`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:47,691::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:47,691::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:47,691::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:47,692::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:47,692::task::1185::TaskManager.Task::(prepare) Task=`450b0a22-f873-41c2-90b4-a908efdc52fd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:47,692::task::595::TaskManager.Task::(_updateState) Task=`450b0a22-f873-41c2-90b4-a908efdc52fd`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:47,693::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:47,693::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:47,694::task::990::TaskManager.Task::(_decref) Task=`450b0a22-f873-41c2-90b4-a908efdc52fd`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:48,698::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:48,699::task::595::TaskManager.Task::(_updateState) Task=`1dfa1d5d-14f4-440c-b366-9a35c51e1e67`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:48,700::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:48,700::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:48,700::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:48,701::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:48,701::task::1185::TaskManager.Task::(prepare) Task=`1dfa1d5d-14f4-440c-b366-9a35c51e1e67`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:48,702::task::595::TaskManager.Task::(_updateState) Task=`1dfa1d5d-14f4-440c-b366-9a35c51e1e67`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:48,702::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:48,702::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:48,703::task::990::TaskManager.Task::(_decref) Task=`1dfa1d5d-14f4-440c-b366-9a35c51e1e67`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:49,716::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:49,731::task::595::TaskManager.Task::(_updateState) Task=`dca7c7c0-08c6-492d-b3c8-d71901092327`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:49,738::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:49,752::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:49,754::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:49,754::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:49,755::task::1185::TaskManager.Task::(prepare) Task=`dca7c7c0-08c6-492d-b3c8-d71901092327`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:49,759::task::595::TaskManager.Task::(_updateState) Task=`dca7c7c0-08c6-492d-b3c8-d71901092327`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:49,765::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:49,767::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:49,772::task::990::TaskManager.Task::(_decref) Task=`dca7c7c0-08c6-492d-b3c8-d71901092327`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:50,788::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:50,789::task::595::TaskManager.Task::(_updateState) Task=`9347db3a-c612-41b9-a49b-0974340cbc38`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:50,789::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:50,790::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:50,791::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:50,791::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:50,792::task::1185::TaskManager.Task::(prepare) Task=`9347db3a-c612-41b9-a49b-0974340cbc38`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:50,793::task::595::TaskManager.Task::(_updateState) Task=`9347db3a-c612-41b9-a49b-0974340cbc38`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:50,794::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:50,794::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:50,794::task::990::TaskManager.Task::(_decref) Task=`9347db3a-c612-41b9-a49b-0974340cbc38`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:51,800::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:51,802::task::595::TaskManager.Task::(_updateState) Task=`09400b28-4407-453c-ab88-cd1ebe1c67c8`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:51,803::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:51,804::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:51,805::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:51,806::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:51,807::task::1185::TaskManager.Task::(prepare) Task=`09400b28-4407-453c-ab88-cd1ebe1c67c8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:51,808::task::595::TaskManager.Task::(_updateState) Task=`09400b28-4407-453c-ab88-cd1ebe1c67c8`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:51,809::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:51,811::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:51,812::task::990::TaskManager.Task::(_decref) Task=`09400b28-4407-453c-ab88-cd1ebe1c67c8`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:52,825::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:52,834::task::595::TaskManager.Task::(_updateState) Task=`26814564-c837-4e71-aae1-7b720279e01e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:52,842::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:52,847::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:52,853::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:52,864::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:52,870::task::1185::TaskManager.Task::(prepare) Task=`26814564-c837-4e71-aae1-7b720279e01e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:52,874::task::595::TaskManager.Task::(_updateState) Task=`26814564-c837-4e71-aae1-7b720279e01e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:52,880::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:52,887::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:52,896::task::990::TaskManager.Task::(_decref) Task=`26814564-c837-4e71-aae1-7b720279e01e`::ref 0 aborting False >Thread-223::WARNING::2014-04-11 22:17:53,346::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-223::DEBUG::2014-04-11 22:17:53,402::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-224::WARNING::2014-04-11 22:17:53,387::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-226::WARNING::2014-04-11 22:17:53,392::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-224::DEBUG::2014-04-11 22:17:53,454::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-225::WARNING::2014-04-11 22:17:53,392::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-226::DEBUG::2014-04-11 22:17:53,485::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-225::DEBUG::2014-04-11 22:17:53,513::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-235::WARNING::2014-04-11 22:17:53,584::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-235::DEBUG::2014-04-11 22:17:53,645::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-236::WARNING::2014-04-11 22:17:53,621::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-237::WARNING::2014-04-11 22:17:53,630::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-236::DEBUG::2014-04-11 22:17:53,680::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-238::WARNING::2014-04-11 22:17:53,671::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-237::DEBUG::2014-04-11 22:17:53,701::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-241::WARNING::2014-04-11 22:17:53,699::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-238::DEBUG::2014-04-11 22:17:53,722::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-242::WARNING::2014-04-11 22:17:53,700::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-241::DEBUG::2014-04-11 22:17:53,742::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-229::ERROR::2014-04-11 22:17:53,750::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',) >Thread-243::WARNING::2014-04-11 22:17:53,719::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-244::WARNING::2014-04-11 22:17:53,741::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-234::ERROR::2014-04-11 22:17:53,806::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 21619c8e-99ea-4813-be02-d708971e5393 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',) >Thread-65::DEBUG::2014-04-11 22:17:53,827::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-28::DEBUG::2014-04-11 22:17:54,035::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:54,127::task::595::TaskManager.Task::(_updateState) Task=`7c58d362-4a74-4bfb-9033-57b6d1f3aff7`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:54,148::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:54,159::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:54,162::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:54,165::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:54,168::task::1185::TaskManager.Task::(prepare) Task=`7c58d362-4a74-4bfb-9033-57b6d1f3aff7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-65::DEBUG::2014-04-11 22:17:54,170::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n754 bytes (754 B) copied, 0.000423611 s, 1.8 MB/s\n'; <rc> = 0 >Thread-28::DEBUG::2014-04-11 22:17:54,171::task::595::TaskManager.Task::(_updateState) Task=`7c58d362-4a74-4bfb-9033-57b6d1f3aff7`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:54,179::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:54,182::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:54,185::task::990::TaskManager.Task::(_decref) Task=`7c58d362-4a74-4bfb-9033-57b6d1f3aff7`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:55,202::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:55,205::task::595::TaskManager.Task::(_updateState) Task=`efdda0af-bfaa-4535-82ba-655b811acf87`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:55,205::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:55,206::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:55,206::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:55,208::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:55,208::task::1185::TaskManager.Task::(prepare) Task=`efdda0af-bfaa-4535-82ba-655b811acf87`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:55,209::task::595::TaskManager.Task::(_updateState) Task=`efdda0af-bfaa-4535-82ba-655b811acf87`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:55,210::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:55,210::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:55,211::task::990::TaskManager.Task::(_decref) Task=`efdda0af-bfaa-4535-82ba-655b811acf87`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:56,216::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:56,217::task::595::TaskManager.Task::(_updateState) Task=`86ceb74a-e6a7-4211-b20e-82beed601fc9`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:56,217::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:56,218::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:56,218::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:56,219::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:56,219::task::1185::TaskManager.Task::(prepare) Task=`86ceb74a-e6a7-4211-b20e-82beed601fc9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:56,219::task::595::TaskManager.Task::(_updateState) Task=`86ceb74a-e6a7-4211-b20e-82beed601fc9`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:56,220::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:56,220::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:56,221::task::990::TaskManager.Task::(_decref) Task=`86ceb74a-e6a7-4211-b20e-82beed601fc9`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:57,230::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:57,230::task::595::TaskManager.Task::(_updateState) Task=`4c8fd0bc-0abf-4bac-85a2-a101deb9dfe5`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:57,231::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:57,231::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:57,231::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:57,232::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:57,232::task::1185::TaskManager.Task::(prepare) Task=`4c8fd0bc-0abf-4bac-85a2-a101deb9dfe5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:57,232::task::595::TaskManager.Task::(_updateState) Task=`4c8fd0bc-0abf-4bac-85a2-a101deb9dfe5`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:57,233::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:57,233::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:57,234::task::990::TaskManager.Task::(_decref) Task=`4c8fd0bc-0abf-4bac-85a2-a101deb9dfe5`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:58,239::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:58,239::task::595::TaskManager.Task::(_updateState) Task=`def69519-df72-46d4-81ce-9bef3b98c82e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:58,240::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:58,240::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:58,241::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:58,241::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:58,241::task::1185::TaskManager.Task::(prepare) Task=`def69519-df72-46d4-81ce-9bef3b98c82e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:58,242::task::595::TaskManager.Task::(_updateState) Task=`def69519-df72-46d4-81ce-9bef3b98c82e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:58,242::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:58,242::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:58,243::task::990::TaskManager.Task::(_decref) Task=`def69519-df72-46d4-81ce-9bef3b98c82e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:17:59,248::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:17:59,249::task::595::TaskManager.Task::(_updateState) Task=`e8286468-c220-4948-aac7-6d92bfcefc72`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:17:59,250::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:17:59,250::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:17:59,251::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:17:59,252::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:59,252::task::1185::TaskManager.Task::(prepare) Task=`e8286468-c220-4948-aac7-6d92bfcefc72`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:17:59,253::task::595::TaskManager.Task::(_updateState) Task=`e8286468-c220-4948-aac7-6d92bfcefc72`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:17:59,254::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:17:59,254::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:17:59,254::task::990::TaskManager.Task::(_decref) Task=`e8286468-c220-4948-aac7-6d92bfcefc72`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:00,263::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:00,264::task::595::TaskManager.Task::(_updateState) Task=`0bd681a3-ed3f-48d1-826d-4789a60cca00`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:00,264::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:00,265::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:00,265::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:00,266::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:00,266::task::1185::TaskManager.Task::(prepare) Task=`0bd681a3-ed3f-48d1-826d-4789a60cca00`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:00,266::task::595::TaskManager.Task::(_updateState) Task=`0bd681a3-ed3f-48d1-826d-4789a60cca00`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:00,267::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:00,267::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:00,267::task::990::TaskManager.Task::(_decref) Task=`0bd681a3-ed3f-48d1-826d-4789a60cca00`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:01,274::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:01,276::task::595::TaskManager.Task::(_updateState) Task=`cbe2f521-5186-4a8e-ad60-4ad16174bf0f`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:01,276::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:01,277::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:01,277::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:01,278::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:01,278::task::1185::TaskManager.Task::(prepare) Task=`cbe2f521-5186-4a8e-ad60-4ad16174bf0f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:01,278::task::595::TaskManager.Task::(_updateState) Task=`cbe2f521-5186-4a8e-ad60-4ad16174bf0f`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:01,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:01,279::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:01,279::task::990::TaskManager.Task::(_decref) Task=`cbe2f521-5186-4a8e-ad60-4ad16174bf0f`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:02,283::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:02,296::task::595::TaskManager.Task::(_updateState) Task=`39c621cd-fd99-4044-a32c-4a8eed46546c`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:02,297::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:02,298::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:02,298::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:02,298::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:02,299::task::1185::TaskManager.Task::(prepare) Task=`39c621cd-fd99-4044-a32c-4a8eed46546c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:02,299::task::595::TaskManager.Task::(_updateState) Task=`39c621cd-fd99-4044-a32c-4a8eed46546c`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:02,300::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:02,300::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:02,301::task::990::TaskManager.Task::(_decref) Task=`39c621cd-fd99-4044-a32c-4a8eed46546c`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:03,305::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:03,306::task::595::TaskManager.Task::(_updateState) Task=`ab9ef7c4-262a-4e10-ac6f-11cdc9e1a8c3`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:03,307::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:03,307::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:03,307::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:03,308::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:03,308::task::1185::TaskManager.Task::(prepare) Task=`ab9ef7c4-262a-4e10-ac6f-11cdc9e1a8c3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:03,309::task::595::TaskManager.Task::(_updateState) Task=`ab9ef7c4-262a-4e10-ac6f-11cdc9e1a8c3`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:03,309::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:03,309::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:03,310::task::990::TaskManager.Task::(_decref) Task=`ab9ef7c4-262a-4e10-ac6f-11cdc9e1a8c3`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:04,315::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:04,323::task::595::TaskManager.Task::(_updateState) Task=`c0541a3c-16cb-41aa-832a-2142f42e4aa8`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:04,324::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:04,325::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:04,325::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:04,326::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:04,326::task::1185::TaskManager.Task::(prepare) Task=`c0541a3c-16cb-41aa-832a-2142f42e4aa8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:04,330::task::595::TaskManager.Task::(_updateState) Task=`c0541a3c-16cb-41aa-832a-2142f42e4aa8`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:04,331::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:04,331::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:04,332::task::990::TaskManager.Task::(_decref) Task=`c0541a3c-16cb-41aa-832a-2142f42e4aa8`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:05,339::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:05,342::task::595::TaskManager.Task::(_updateState) Task=`5111a9d0-a9da-4e00-82d1-607535153de7`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:05,342::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:05,343::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:05,343::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:05,347::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:05,349::task::1185::TaskManager.Task::(prepare) Task=`5111a9d0-a9da-4e00-82d1-607535153de7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:05,349::task::595::TaskManager.Task::(_updateState) Task=`5111a9d0-a9da-4e00-82d1-607535153de7`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:05,349::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:05,350::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:05,354::task::990::TaskManager.Task::(_decref) Task=`5111a9d0-a9da-4e00-82d1-607535153de7`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:06,360::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:06,361::task::595::TaskManager.Task::(_updateState) Task=`314a9b83-4d11-4b5a-bc83-40a33dc95327`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:06,361::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:06,362::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:06,362::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:06,363::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:06,363::task::1185::TaskManager.Task::(prepare) Task=`314a9b83-4d11-4b5a-bc83-40a33dc95327`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:06,364::task::595::TaskManager.Task::(_updateState) Task=`314a9b83-4d11-4b5a-bc83-40a33dc95327`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:06,364::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:06,365::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:06,365::task::990::TaskManager.Task::(_decref) Task=`314a9b83-4d11-4b5a-bc83-40a33dc95327`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:07,369::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:07,371::task::595::TaskManager.Task::(_updateState) Task=`b2b03a9a-0160-467d-a438-7cdff90b1e70`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:07,371::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:07,371::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:07,372::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:07,372::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:07,373::task::1185::TaskManager.Task::(prepare) Task=`b2b03a9a-0160-467d-a438-7cdff90b1e70`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:07,373::task::595::TaskManager.Task::(_updateState) Task=`b2b03a9a-0160-467d-a438-7cdff90b1e70`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:07,374::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:07,374::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:07,375::task::990::TaskManager.Task::(_decref) Task=`b2b03a9a-0160-467d-a438-7cdff90b1e70`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:08,379::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:08,380::task::595::TaskManager.Task::(_updateState) Task=`86249225-1df5-4531-8155-e980007c71bb`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:08,381::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:08,382::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:08,382::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:08,383::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:08,383::task::1185::TaskManager.Task::(prepare) Task=`86249225-1df5-4531-8155-e980007c71bb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:08,384::task::595::TaskManager.Task::(_updateState) Task=`86249225-1df5-4531-8155-e980007c71bb`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:08,385::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:08,385::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:08,385::task::990::TaskManager.Task::(_decref) Task=`86249225-1df5-4531-8155-e980007c71bb`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:09,431::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:09,473::task::595::TaskManager.Task::(_updateState) Task=`01e9ac2c-80a4-4f23-8355-5170b4cb23da`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:09,510::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:09,536::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:09,564::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:09,595::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:09,625::task::1185::TaskManager.Task::(prepare) Task=`01e9ac2c-80a4-4f23-8355-5170b4cb23da`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:09,658::task::595::TaskManager.Task::(_updateState) Task=`01e9ac2c-80a4-4f23-8355-5170b4cb23da`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:09,688::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:09,720::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:09,758::task::990::TaskManager.Task::(_decref) Task=`01e9ac2c-80a4-4f23-8355-5170b4cb23da`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:10,930::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:10,996::task::595::TaskManager.Task::(_updateState) Task=`62e623c2-df9c-47c1-82d3-aadf0a39bf24`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:11,050::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:11,090::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:11,132::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:11,171::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:11,213::task::1185::TaskManager.Task::(prepare) Task=`62e623c2-df9c-47c1-82d3-aadf0a39bf24`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:11,257::task::595::TaskManager.Task::(_updateState) Task=`62e623c2-df9c-47c1-82d3-aadf0a39bf24`::moving from state preparing -> state finished >Thread-247::WARNING::2014-04-11 22:18:11,234::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-28::DEBUG::2014-04-11 22:18:11,295::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-248::WARNING::2014-04-11 22:18:11,262::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-247::DEBUG::2014-04-11 22:18:11,307::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-249::WARNING::2014-04-11 22:18:11,270::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-28::DEBUG::2014-04-11 22:18:11,330::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-248::DEBUG::2014-04-11 22:18:11,341::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-250::WARNING::2014-04-11 22:18:11,277::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-249::DEBUG::2014-04-11 22:18:11,366::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-28::DEBUG::2014-04-11 22:18:11,382::task::990::TaskManager.Task::(_decref) Task=`62e623c2-df9c-47c1-82d3-aadf0a39bf24`::ref 0 aborting False >Thread-250::DEBUG::2014-04-11 22:18:11,399::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-228::ERROR::2014-04-11 22:18:11,441::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-254::WARNING::2014-04-11 22:18:11,453::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-72::ERROR::2014-04-11 22:18:11,491::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-255::WARNING::2014-04-11 22:18:11,481::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-72::ERROR::2014-04-11 22:18:11,515::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-254::DEBUG::2014-04-11 22:18:11,518::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-256::WARNING::2014-04-11 22:18:11,495::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-255::DEBUG::2014-04-11 22:18:11,538::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-257::WARNING::2014-04-11 22:18:11,495::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-72::DEBUG::2014-04-11 22:18:11,556::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-256::DEBUG::2014-04-11 22:18:11,559::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-258::WARNING::2014-04-11 22:18:11,541::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-257::DEBUG::2014-04-11 22:18:11,575::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-259::WARNING::2014-04-11 22:18:11,542::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-258::DEBUG::2014-04-11 22:18:11,594::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-260::WARNING::2014-04-11 22:18:11,594::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-259::DEBUG::2014-04-11 22:18:11,610::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-261::WARNING::2014-04-11 22:18:11,603::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-260::DEBUG::2014-04-11 22:18:11,627::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-261::DEBUG::2014-04-11 22:18:11,638::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-230::ERROR::2014-04-11 22:18:11,647::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-265::WARNING::2014-04-11 22:18:11,636::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-232::ERROR::2014-04-11 22:18:11,656::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-72::DEBUG::2014-04-11 22:18:11,657::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None) >Thread-74::ERROR::2014-04-11 22:18:11,670::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-265::DEBUG::2014-04-11 22:18:11,672::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-268::DEBUG::2014-04-11 22:18:11,672::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-266::WARNING::2014-04-11 22:18:11,657::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-82::ERROR::2014-04-11 22:18:11,692::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-74::ERROR::2014-04-11 22:18:11,700::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-267::WARNING::2014-04-11 22:18:11,672::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-82::ERROR::2014-04-11 22:18:11,715::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-266::DEBUG::2014-04-11 22:18:11,715::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-268::WARNING::2014-04-11 22:18:11,712::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-267::DEBUG::2014-04-11 22:18:11,723::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-74::DEBUG::2014-04-11 22:18:11,724::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-268::DEBUG::2014-04-11 22:18:11,728::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-82::DEBUG::2014-04-11 22:18:11,736::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-231::ERROR::2014-04-11 22:18:11,736::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-81::ERROR::2014-04-11 22:18:11,745::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-81::ERROR::2014-04-11 22:18:11,750::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-74::DEBUG::2014-04-11 22:18:11,757::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None) >Thread-82::DEBUG::2014-04-11 22:18:11,758::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-81::DEBUG::2014-04-11 22:18:11,758::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-81::DEBUG::2014-04-11 22:18:11,782::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-72::DEBUG::2014-04-11 22:18:12,013::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5 >Thread-81::DEBUG::2014-04-11 22:18:12,024::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-72::WARNING::2014-04-11 22:18:12,026::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8'] >Thread-81::WARNING::2014-04-11 22:18:12,026::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-74::DEBUG::2014-04-11 22:18:12,155::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5 >Thread-82::DEBUG::2014-04-11 22:18:12,243::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-74::WARNING::2014-04-11 22:18:12,274::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce'] >Thread-82::WARNING::2014-04-11 22:18:12,274::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-82::DEBUG::2014-04-11 22:18:12,283::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:18:12,532::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:12,538::task::595::TaskManager.Task::(_updateState) Task=`92dcd58a-1b57-4dd3-9584-5fe0f05ce5f6`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:12,539::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:12,540::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:12,540::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:12,541::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:12,542::task::1185::TaskManager.Task::(prepare) Task=`92dcd58a-1b57-4dd3-9584-5fe0f05ce5f6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:12,542::task::595::TaskManager.Task::(_updateState) Task=`92dcd58a-1b57-4dd3-9584-5fe0f05ce5f6`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:12,543::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:12,546::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:12,547::task::990::TaskManager.Task::(_decref) Task=`92dcd58a-1b57-4dd3-9584-5fe0f05ce5f6`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:13,553::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:13,554::task::595::TaskManager.Task::(_updateState) Task=`dc34b025-50a2-4e91-ae1b-78a201df0490`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:13,554::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:13,554::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:13,555::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:13,555::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:13,555::task::1185::TaskManager.Task::(prepare) Task=`dc34b025-50a2-4e91-ae1b-78a201df0490`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:13,556::task::595::TaskManager.Task::(_updateState) Task=`dc34b025-50a2-4e91-ae1b-78a201df0490`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:13,556::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:13,556::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:13,557::task::990::TaskManager.Task::(_decref) Task=`dc34b025-50a2-4e91-ae1b-78a201df0490`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:14,563::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:14,565::task::595::TaskManager.Task::(_updateState) Task=`c7e15481-fb87-4420-8088-9fd841a79db0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:14,565::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:14,566::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:14,566::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:14,567::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:14,567::task::1185::TaskManager.Task::(prepare) Task=`c7e15481-fb87-4420-8088-9fd841a79db0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:14,567::task::595::TaskManager.Task::(_updateState) Task=`c7e15481-fb87-4420-8088-9fd841a79db0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:14,568::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:14,568::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:14,568::task::990::TaskManager.Task::(_decref) Task=`c7e15481-fb87-4420-8088-9fd841a79db0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:15,574::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:15,575::task::595::TaskManager.Task::(_updateState) Task=`cf8df286-c10f-45fb-b06c-dcacbdc8f0d1`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:15,575::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:15,575::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:15,575::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:15,576::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:15,576::task::1185::TaskManager.Task::(prepare) Task=`cf8df286-c10f-45fb-b06c-dcacbdc8f0d1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:15,576::task::595::TaskManager.Task::(_updateState) Task=`cf8df286-c10f-45fb-b06c-dcacbdc8f0d1`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:15,577::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:15,577::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:15,577::task::990::TaskManager.Task::(_decref) Task=`cf8df286-c10f-45fb-b06c-dcacbdc8f0d1`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:16,581::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:16,582::task::595::TaskManager.Task::(_updateState) Task=`9ef17120-c0f9-4d5a-88de-1ee52bffec2e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:16,582::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:16,582::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:16,583::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:16,583::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:16,583::task::1185::TaskManager.Task::(prepare) Task=`9ef17120-c0f9-4d5a-88de-1ee52bffec2e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:16,584::task::595::TaskManager.Task::(_updateState) Task=`9ef17120-c0f9-4d5a-88de-1ee52bffec2e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:16,584::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:16,584::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:16,584::task::990::TaskManager.Task::(_decref) Task=`9ef17120-c0f9-4d5a-88de-1ee52bffec2e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:17,588::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:17,589::task::595::TaskManager.Task::(_updateState) Task=`f7ee35f1-31c3-4247-8409-bc2aed6be144`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:17,589::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:17,590::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:17,590::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:17,590::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:17,591::task::1185::TaskManager.Task::(prepare) Task=`f7ee35f1-31c3-4247-8409-bc2aed6be144`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:17,591::task::595::TaskManager.Task::(_updateState) Task=`f7ee35f1-31c3-4247-8409-bc2aed6be144`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:17,591::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:17,591::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:17,592::task::990::TaskManager.Task::(_decref) Task=`f7ee35f1-31c3-4247-8409-bc2aed6be144`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:18,596::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:18,596::task::595::TaskManager.Task::(_updateState) Task=`345a78f4-4b1c-46ff-b67a-7865a8181ec7`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:18,597::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:18,597::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:18,597::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:18,597::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:18,598::task::1185::TaskManager.Task::(prepare) Task=`345a78f4-4b1c-46ff-b67a-7865a8181ec7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:18,598::task::595::TaskManager.Task::(_updateState) Task=`345a78f4-4b1c-46ff-b67a-7865a8181ec7`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:18,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:18,599::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:18,599::task::990::TaskManager.Task::(_decref) Task=`345a78f4-4b1c-46ff-b67a-7865a8181ec7`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:19,604::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:19,605::task::595::TaskManager.Task::(_updateState) Task=`03158728-e18b-4576-88e8-a723cb9da9c0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:19,606::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:19,606::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:19,606::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:19,607::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:19,607::task::1185::TaskManager.Task::(prepare) Task=`03158728-e18b-4576-88e8-a723cb9da9c0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:19,608::task::595::TaskManager.Task::(_updateState) Task=`03158728-e18b-4576-88e8-a723cb9da9c0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:19,609::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:19,609::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:19,609::task::990::TaskManager.Task::(_decref) Task=`03158728-e18b-4576-88e8-a723cb9da9c0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:20,615::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:20,616::task::595::TaskManager.Task::(_updateState) Task=`ffeacdd2-0f93-450b-8472-7523089bb602`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:20,616::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:20,617::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:20,617::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:20,617::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:20,618::task::1185::TaskManager.Task::(prepare) Task=`ffeacdd2-0f93-450b-8472-7523089bb602`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:20,618::task::595::TaskManager.Task::(_updateState) Task=`ffeacdd2-0f93-450b-8472-7523089bb602`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:20,618::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:20,619::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:20,619::task::990::TaskManager.Task::(_decref) Task=`ffeacdd2-0f93-450b-8472-7523089bb602`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:21,623::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:21,623::task::595::TaskManager.Task::(_updateState) Task=`cc944da9-0bb7-45b5-a817-3c3e7bb3fbe5`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:21,624::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:21,624::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:21,624::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:21,625::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:21,625::task::1185::TaskManager.Task::(prepare) Task=`cc944da9-0bb7-45b5-a817-3c3e7bb3fbe5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:21,625::task::595::TaskManager.Task::(_updateState) Task=`cc944da9-0bb7-45b5-a817-3c3e7bb3fbe5`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:21,626::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:21,626::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:21,626::task::990::TaskManager.Task::(_decref) Task=`cc944da9-0bb7-45b5-a817-3c3e7bb3fbe5`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:22,630::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:22,632::task::595::TaskManager.Task::(_updateState) Task=`6433e4a0-66f7-457a-ba30-a06889966dde`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:22,633::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:22,633::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:22,633::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:22,634::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:22,634::task::1185::TaskManager.Task::(prepare) Task=`6433e4a0-66f7-457a-ba30-a06889966dde`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:22,634::task::595::TaskManager.Task::(_updateState) Task=`6433e4a0-66f7-457a-ba30-a06889966dde`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:22,635::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:22,635::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:22,635::task::990::TaskManager.Task::(_decref) Task=`6433e4a0-66f7-457a-ba30-a06889966dde`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:23,639::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:23,640::task::595::TaskManager.Task::(_updateState) Task=`5d650740-2c0e-4fde-91fb-8e7f845fc0c4`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:23,640::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:23,641::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:23,641::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:23,641::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:23,642::task::1185::TaskManager.Task::(prepare) Task=`5d650740-2c0e-4fde-91fb-8e7f845fc0c4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:23,642::task::595::TaskManager.Task::(_updateState) Task=`5d650740-2c0e-4fde-91fb-8e7f845fc0c4`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:23,642::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:23,643::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:23,643::task::990::TaskManager.Task::(_decref) Task=`5d650740-2c0e-4fde-91fb-8e7f845fc0c4`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:24,648::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:24,650::task::595::TaskManager.Task::(_updateState) Task=`d89c7e60-68e5-4e1e-93ba-e383421d3b85`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:24,650::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:24,651::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:24,651::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:24,652::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:24,652::task::1185::TaskManager.Task::(prepare) Task=`d89c7e60-68e5-4e1e-93ba-e383421d3b85`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:24,653::task::595::TaskManager.Task::(_updateState) Task=`d89c7e60-68e5-4e1e-93ba-e383421d3b85`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:24,653::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:24,653::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:24,654::task::990::TaskManager.Task::(_decref) Task=`d89c7e60-68e5-4e1e-93ba-e383421d3b85`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:25,660::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:25,661::task::595::TaskManager.Task::(_updateState) Task=`de9bd4d3-cd3a-41df-9b79-b99d90239ac4`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:25,662::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:25,662::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:25,663::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:25,663::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:25,663::task::1185::TaskManager.Task::(prepare) Task=`de9bd4d3-cd3a-41df-9b79-b99d90239ac4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:25,664::task::595::TaskManager.Task::(_updateState) Task=`de9bd4d3-cd3a-41df-9b79-b99d90239ac4`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:25,664::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:25,665::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:25,665::task::990::TaskManager.Task::(_decref) Task=`de9bd4d3-cd3a-41df-9b79-b99d90239ac4`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:26,670::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:26,671::task::595::TaskManager.Task::(_updateState) Task=`c19a6aee-2c3c-42c4-8e24-0143a0cbd0b3`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:26,671::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:26,671::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:26,672::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:26,673::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:26,673::task::1185::TaskManager.Task::(prepare) Task=`c19a6aee-2c3c-42c4-8e24-0143a0cbd0b3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:26,674::task::595::TaskManager.Task::(_updateState) Task=`c19a6aee-2c3c-42c4-8e24-0143a0cbd0b3`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:26,674::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:26,674::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:26,675::task::990::TaskManager.Task::(_decref) Task=`c19a6aee-2c3c-42c4-8e24-0143a0cbd0b3`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:27,679::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:27,683::task::595::TaskManager.Task::(_updateState) Task=`87596d5b-080f-4e8a-9bb8-03a4cb247c10`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:27,684::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:27,684::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:27,684::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:27,685::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:27,685::task::1185::TaskManager.Task::(prepare) Task=`87596d5b-080f-4e8a-9bb8-03a4cb247c10`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:27,686::task::595::TaskManager.Task::(_updateState) Task=`87596d5b-080f-4e8a-9bb8-03a4cb247c10`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:27,686::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:27,686::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:27,686::task::990::TaskManager.Task::(_decref) Task=`87596d5b-080f-4e8a-9bb8-03a4cb247c10`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:28,691::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:28,691::task::595::TaskManager.Task::(_updateState) Task=`10085ea4-4aeb-4999-acc2-2c37f9851039`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:28,692::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:28,692::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:28,692::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:28,693::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:28,693::task::1185::TaskManager.Task::(prepare) Task=`10085ea4-4aeb-4999-acc2-2c37f9851039`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:28,693::task::595::TaskManager.Task::(_updateState) Task=`10085ea4-4aeb-4999-acc2-2c37f9851039`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:28,694::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:28,694::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:28,694::task::990::TaskManager.Task::(_decref) Task=`10085ea4-4aeb-4999-acc2-2c37f9851039`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:29,700::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:29,702::task::595::TaskManager.Task::(_updateState) Task=`3d465e22-26a1-4849-bac9-327c21f85c8f`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:29,704::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:29,704::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:29,704::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:29,705::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:29,705::task::1185::TaskManager.Task::(prepare) Task=`3d465e22-26a1-4849-bac9-327c21f85c8f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:29,706::task::595::TaskManager.Task::(_updateState) Task=`3d465e22-26a1-4849-bac9-327c21f85c8f`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:29,706::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:29,707::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:29,707::task::990::TaskManager.Task::(_decref) Task=`3d465e22-26a1-4849-bac9-327c21f85c8f`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:30,712::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:30,713::task::595::TaskManager.Task::(_updateState) Task=`725f09ab-e5a5-4c5c-8b2d-3a3e1805b2a1`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:30,713::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:30,714::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:30,714::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:30,714::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:30,715::task::1185::TaskManager.Task::(prepare) Task=`725f09ab-e5a5-4c5c-8b2d-3a3e1805b2a1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:30,715::task::595::TaskManager.Task::(_updateState) Task=`725f09ab-e5a5-4c5c-8b2d-3a3e1805b2a1`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:30,715::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:30,716::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:30,716::task::990::TaskManager.Task::(_decref) Task=`725f09ab-e5a5-4c5c-8b2d-3a3e1805b2a1`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:31,720::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:31,721::task::595::TaskManager.Task::(_updateState) Task=`6728b7bf-04b5-4de1-8b82-77c49883b322`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:31,721::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:31,722::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:31,722::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:31,722::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:31,723::task::1185::TaskManager.Task::(prepare) Task=`6728b7bf-04b5-4de1-8b82-77c49883b322`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:31,723::task::595::TaskManager.Task::(_updateState) Task=`6728b7bf-04b5-4de1-8b82-77c49883b322`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:31,723::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:31,724::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:31,724::task::990::TaskManager.Task::(_decref) Task=`6728b7bf-04b5-4de1-8b82-77c49883b322`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:32,728::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:32,731::task::595::TaskManager.Task::(_updateState) Task=`b889178c-db88-4671-91d2-ac521cc43147`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:32,731::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:32,732::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:32,732::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:32,732::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:32,732::task::1185::TaskManager.Task::(prepare) Task=`b889178c-db88-4671-91d2-ac521cc43147`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:32,733::task::595::TaskManager.Task::(_updateState) Task=`b889178c-db88-4671-91d2-ac521cc43147`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:32,733::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:32,733::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:32,734::task::990::TaskManager.Task::(_decref) Task=`b889178c-db88-4671-91d2-ac521cc43147`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:33,738::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:33,739::task::595::TaskManager.Task::(_updateState) Task=`a10248c0-e8bc-4dc9-883b-e79759e087da`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:33,739::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:33,739::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:33,740::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:33,740::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:33,740::task::1185::TaskManager.Task::(prepare) Task=`a10248c0-e8bc-4dc9-883b-e79759e087da`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:33,741::task::595::TaskManager.Task::(_updateState) Task=`a10248c0-e8bc-4dc9-883b-e79759e087da`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:33,741::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:33,741::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:33,742::task::990::TaskManager.Task::(_decref) Task=`a10248c0-e8bc-4dc9-883b-e79759e087da`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:34,747::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:34,748::task::595::TaskManager.Task::(_updateState) Task=`b7523cd7-d7e1-49e3-982f-96c6c2e50111`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:34,749::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:34,749::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:34,750::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:34,750::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:34,751::task::1185::TaskManager.Task::(prepare) Task=`b7523cd7-d7e1-49e3-982f-96c6c2e50111`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:34,752::task::595::TaskManager.Task::(_updateState) Task=`b7523cd7-d7e1-49e3-982f-96c6c2e50111`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:34,753::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:34,754::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:34,755::task::990::TaskManager.Task::(_decref) Task=`b7523cd7-d7e1-49e3-982f-96c6c2e50111`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:35,761::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:35,762::task::595::TaskManager.Task::(_updateState) Task=`ff79f9d6-a9a6-401f-bd99-8f709c37209e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:35,763::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:35,763::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:35,764::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:35,765::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:35,765::task::1185::TaskManager.Task::(prepare) Task=`ff79f9d6-a9a6-401f-bd99-8f709c37209e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:35,765::task::595::TaskManager.Task::(_updateState) Task=`ff79f9d6-a9a6-401f-bd99-8f709c37209e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:35,766::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:35,766::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:35,769::task::990::TaskManager.Task::(_decref) Task=`ff79f9d6-a9a6-401f-bd99-8f709c37209e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:36,774::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:36,774::task::595::TaskManager.Task::(_updateState) Task=`100a6adf-f783-4bb8-9249-61fe49cee090`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:36,775::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:36,775::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:36,775::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:36,775::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:36,776::task::1185::TaskManager.Task::(prepare) Task=`100a6adf-f783-4bb8-9249-61fe49cee090`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:36,776::task::595::TaskManager.Task::(_updateState) Task=`100a6adf-f783-4bb8-9249-61fe49cee090`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:36,777::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:36,777::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:36,777::task::990::TaskManager.Task::(_decref) Task=`100a6adf-f783-4bb8-9249-61fe49cee090`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:37,782::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:37,798::task::595::TaskManager.Task::(_updateState) Task=`a2a9afe1-95e7-4776-b20f-8ed0446d0f22`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:37,799::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:37,799::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:37,800::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:37,800::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:37,801::task::1185::TaskManager.Task::(prepare) Task=`a2a9afe1-95e7-4776-b20f-8ed0446d0f22`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:37,801::task::595::TaskManager.Task::(_updateState) Task=`a2a9afe1-95e7-4776-b20f-8ed0446d0f22`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:37,801::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:37,802::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:37,802::task::990::TaskManager.Task::(_decref) Task=`a2a9afe1-95e7-4776-b20f-8ed0446d0f22`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:38,806::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:38,807::task::595::TaskManager.Task::(_updateState) Task=`67fe117d-2fab-4730-8e33-f7dd0e7ae45e`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:38,808::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:38,808::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:38,808::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:38,809::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:38,809::task::1185::TaskManager.Task::(prepare) Task=`67fe117d-2fab-4730-8e33-f7dd0e7ae45e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:38,809::task::595::TaskManager.Task::(_updateState) Task=`67fe117d-2fab-4730-8e33-f7dd0e7ae45e`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:38,809::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:38,810::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:38,810::task::990::TaskManager.Task::(_decref) Task=`67fe117d-2fab-4730-8e33-f7dd0e7ae45e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:39,824::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:39,826::task::595::TaskManager.Task::(_updateState) Task=`9572a15f-cb81-436a-8468-1d8d61b38470`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:39,826::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:39,826::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:39,827::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:39,828::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:39,828::task::1185::TaskManager.Task::(prepare) Task=`9572a15f-cb81-436a-8468-1d8d61b38470`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:39,829::task::595::TaskManager.Task::(_updateState) Task=`9572a15f-cb81-436a-8468-1d8d61b38470`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:39,829::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:39,829::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:39,830::task::990::TaskManager.Task::(_decref) Task=`9572a15f-cb81-436a-8468-1d8d61b38470`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:40,837::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:40,838::task::595::TaskManager.Task::(_updateState) Task=`3df8b062-8349-477d-81a0-efeea41371b6`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:40,838::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:40,840::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:40,840::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:40,840::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:40,841::task::1185::TaskManager.Task::(prepare) Task=`3df8b062-8349-477d-81a0-efeea41371b6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:40,842::task::595::TaskManager.Task::(_updateState) Task=`3df8b062-8349-477d-81a0-efeea41371b6`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:40,843::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:40,843::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:40,844::task::990::TaskManager.Task::(_decref) Task=`3df8b062-8349-477d-81a0-efeea41371b6`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:41,848::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:41,860::task::595::TaskManager.Task::(_updateState) Task=`afa34c1b-7894-492f-aa51-2cdf6e4f4527`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:41,860::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:41,860::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:41,861::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:41,861::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:41,861::task::1185::TaskManager.Task::(prepare) Task=`afa34c1b-7894-492f-aa51-2cdf6e4f4527`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:41,862::task::595::TaskManager.Task::(_updateState) Task=`afa34c1b-7894-492f-aa51-2cdf6e4f4527`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:41,862::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:41,862::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:41,863::task::990::TaskManager.Task::(_decref) Task=`afa34c1b-7894-492f-aa51-2cdf6e4f4527`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:42,868::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:42,880::task::595::TaskManager.Task::(_updateState) Task=`add4e5cd-2924-48b0-ae51-ea58aeeb72d2`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:42,881::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:42,881::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:42,881::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:42,882::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:42,882::task::1185::TaskManager.Task::(prepare) Task=`add4e5cd-2924-48b0-ae51-ea58aeeb72d2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:42,882::task::595::TaskManager.Task::(_updateState) Task=`add4e5cd-2924-48b0-ae51-ea58aeeb72d2`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:42,883::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:42,883::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:42,883::task::990::TaskManager.Task::(_decref) Task=`add4e5cd-2924-48b0-ae51-ea58aeeb72d2`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:43,888::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:43,889::task::595::TaskManager.Task::(_updateState) Task=`8e163546-20f8-4312-a0a8-35dc5748a107`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:43,889::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:43,889::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:43,889::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:43,890::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:43,890::task::1185::TaskManager.Task::(prepare) Task=`8e163546-20f8-4312-a0a8-35dc5748a107`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:43,890::task::595::TaskManager.Task::(_updateState) Task=`8e163546-20f8-4312-a0a8-35dc5748a107`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:43,891::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:43,891::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:43,892::task::990::TaskManager.Task::(_decref) Task=`8e163546-20f8-4312-a0a8-35dc5748a107`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:44,897::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:44,898::task::595::TaskManager.Task::(_updateState) Task=`a7edda22-dfbd-4882-9020-7d549f349877`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:44,899::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:44,899::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:44,899::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:44,900::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:44,900::task::1185::TaskManager.Task::(prepare) Task=`a7edda22-dfbd-4882-9020-7d549f349877`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:44,900::task::595::TaskManager.Task::(_updateState) Task=`a7edda22-dfbd-4882-9020-7d549f349877`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:44,901::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:44,901::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:44,901::task::990::TaskManager.Task::(_decref) Task=`a7edda22-dfbd-4882-9020-7d549f349877`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:45,906::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:45,907::task::595::TaskManager.Task::(_updateState) Task=`4bc4ced6-307b-42cd-b5bb-8dece2463fb0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:45,908::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:45,908::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:45,908::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:45,909::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:45,909::task::1185::TaskManager.Task::(prepare) Task=`4bc4ced6-307b-42cd-b5bb-8dece2463fb0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:45,909::task::595::TaskManager.Task::(_updateState) Task=`4bc4ced6-307b-42cd-b5bb-8dece2463fb0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:45,910::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:45,910::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:45,910::task::990::TaskManager.Task::(_decref) Task=`4bc4ced6-307b-42cd-b5bb-8dece2463fb0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:46,915::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:46,915::task::595::TaskManager.Task::(_updateState) Task=`63dbaf1f-a8ab-4b4c-b0f1-dc66ea918d76`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:46,916::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:46,916::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:46,916::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:46,917::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:46,917::task::1185::TaskManager.Task::(prepare) Task=`63dbaf1f-a8ab-4b4c-b0f1-dc66ea918d76`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:46,918::task::595::TaskManager.Task::(_updateState) Task=`63dbaf1f-a8ab-4b4c-b0f1-dc66ea918d76`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:46,918::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:46,918::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:46,919::task::990::TaskManager.Task::(_decref) Task=`63dbaf1f-a8ab-4b4c-b0f1-dc66ea918d76`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:47,936::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:47,936::task::595::TaskManager.Task::(_updateState) Task=`786bb5f7-ebf6-4dd7-9790-d74d01c09cb0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:47,937::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:47,937::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:47,937::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:47,938::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:47,938::task::1185::TaskManager.Task::(prepare) Task=`786bb5f7-ebf6-4dd7-9790-d74d01c09cb0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:47,938::task::595::TaskManager.Task::(_updateState) Task=`786bb5f7-ebf6-4dd7-9790-d74d01c09cb0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:47,939::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:47,939::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:47,939::task::990::TaskManager.Task::(_decref) Task=`786bb5f7-ebf6-4dd7-9790-d74d01c09cb0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:48,944::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:48,944::task::595::TaskManager.Task::(_updateState) Task=`6bcecac2-9ee0-4a29-a84e-b119496eaeec`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:48,945::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:48,946::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:48,946::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:48,946::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:48,947::task::1185::TaskManager.Task::(prepare) Task=`6bcecac2-9ee0-4a29-a84e-b119496eaeec`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:48,947::task::595::TaskManager.Task::(_updateState) Task=`6bcecac2-9ee0-4a29-a84e-b119496eaeec`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:48,947::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:48,947::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:48,948::task::990::TaskManager.Task::(_decref) Task=`6bcecac2-9ee0-4a29-a84e-b119496eaeec`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:49,957::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:49,962::task::595::TaskManager.Task::(_updateState) Task=`546b30c9-e006-4f69-a209-af2d3c3041a4`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:49,964::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:49,965::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:49,966::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:49,966::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:49,968::task::1185::TaskManager.Task::(prepare) Task=`546b30c9-e006-4f69-a209-af2d3c3041a4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:49,973::task::595::TaskManager.Task::(_updateState) Task=`546b30c9-e006-4f69-a209-af2d3c3041a4`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:49,977::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:49,978::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:49,978::task::990::TaskManager.Task::(_decref) Task=`546b30c9-e006-4f69-a209-af2d3c3041a4`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:50,986::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:50,987::task::595::TaskManager.Task::(_updateState) Task=`22b79349-d6df-432f-be15-b7a961488c6a`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:50,988::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:50,989::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:50,989::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:50,990::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:50,990::task::1185::TaskManager.Task::(prepare) Task=`22b79349-d6df-432f-be15-b7a961488c6a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:50,990::task::595::TaskManager.Task::(_updateState) Task=`22b79349-d6df-432f-be15-b7a961488c6a`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:50,991::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:50,991::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:50,992::task::990::TaskManager.Task::(_decref) Task=`22b79349-d6df-432f-be15-b7a961488c6a`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:51,997::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:51,998::task::595::TaskManager.Task::(_updateState) Task=`94cbea1b-e831-4ac9-bd42-8f7a0dd89525`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:51,999::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:51,999::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:52,000::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:52,001::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:52,001::task::1185::TaskManager.Task::(prepare) Task=`94cbea1b-e831-4ac9-bd42-8f7a0dd89525`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:52,002::task::595::TaskManager.Task::(_updateState) Task=`94cbea1b-e831-4ac9-bd42-8f7a0dd89525`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:52,002::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:52,003::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:52,004::task::990::TaskManager.Task::(_decref) Task=`94cbea1b-e831-4ac9-bd42-8f7a0dd89525`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:53,010::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:53,014::task::595::TaskManager.Task::(_updateState) Task=`21fd9865-246e-4662-bff7-4c6e4e46df9a`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:53,016::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:53,019::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:53,021::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:53,028::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:53,029::task::1185::TaskManager.Task::(prepare) Task=`21fd9865-246e-4662-bff7-4c6e4e46df9a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:53,031::task::595::TaskManager.Task::(_updateState) Task=`21fd9865-246e-4662-bff7-4c6e4e46df9a`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:53,034::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:53,038::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:53,043::task::990::TaskManager.Task::(_decref) Task=`21fd9865-246e-4662-bff7-4c6e4e46df9a`::ref 0 aborting False >Thread-271::WARNING::2014-04-11 22:18:53,850::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-272::WARNING::2014-04-11 22:18:53,852::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-271::DEBUG::2014-04-11 22:18:53,891::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-272::DEBUG::2014-04-11 22:18:53,909::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-273::WARNING::2014-04-11 22:18:53,952::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-274::WARNING::2014-04-11 22:18:53,953::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-273::DEBUG::2014-04-11 22:18:53,977::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-274::DEBUG::2014-04-11 22:18:53,990::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-73::ERROR::2014-04-11 22:18:54,012::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',) >Thread-73::ERROR::2014-04-11 22:18:54,031::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',) >Thread-28::DEBUG::2014-04-11 22:18:54,072::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-277::WARNING::2014-04-11 22:18:54,081::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-28::DEBUG::2014-04-11 22:18:54,096::task::595::TaskManager.Task::(_updateState) Task=`448a63be-7d18-44f9-ad12-a4af4d7c550e`::moving from state init -> state preparing >Thread-277::DEBUG::2014-04-11 22:18:54,110::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-28::INFO::2014-04-11 22:18:54,117::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:54,132::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-278::WARNING::2014-04-11 22:18:54,125::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-28::DEBUG::2014-04-11 22:18:54,142::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-279::WARNING::2014-04-11 22:18:54,131::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-278::DEBUG::2014-04-11 22:18:54,147::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-280::WARNING::2014-04-11 22:18:54,143::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-28::INFO::2014-04-11 22:18:54,154::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-279::DEBUG::2014-04-11 22:18:54,158::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-280::DEBUG::2014-04-11 22:18:54,165::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-28::DEBUG::2014-04-11 22:18:54,166::task::1185::TaskManager.Task::(prepare) Task=`448a63be-7d18-44f9-ad12-a4af4d7c550e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:54,176::task::595::TaskManager.Task::(_updateState) Task=`448a63be-7d18-44f9-ad12-a4af4d7c550e`::moving from state preparing -> state finished >Thread-83::ERROR::2014-04-11 22:18:54,177::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 21619c8e-99ea-4813-be02-d708971e5393 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',) >Thread-28::DEBUG::2014-04-11 22:18:54,183::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-83::ERROR::2014-04-11 22:18:54,187::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 21619c8e-99ea-4813-be02-d708971e5393 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('21619c8e-99ea-4813-be02-d708971e5393',) >Thread-28::DEBUG::2014-04-11 22:18:54,189::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:54,200::task::990::TaskManager.Task::(_decref) Task=`448a63be-7d18-44f9-ad12-a4af4d7c550e`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:55,211::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:55,213::task::595::TaskManager.Task::(_updateState) Task=`bb0a5da6-ef21-4250-9a01-c468e9cd36dd`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:55,214::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:55,215::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:55,215::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:55,215::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:55,216::task::1185::TaskManager.Task::(prepare) Task=`bb0a5da6-ef21-4250-9a01-c468e9cd36dd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:55,216::task::595::TaskManager.Task::(_updateState) Task=`bb0a5da6-ef21-4250-9a01-c468e9cd36dd`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:55,217::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:55,219::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:55,219::task::990::TaskManager.Task::(_decref) Task=`bb0a5da6-ef21-4250-9a01-c468e9cd36dd`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:56,225::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:56,227::task::595::TaskManager.Task::(_updateState) Task=`c58349f1-1039-49be-b550-ffac9e4ee192`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:56,228::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:56,228::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:56,228::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:56,230::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:56,230::task::1185::TaskManager.Task::(prepare) Task=`c58349f1-1039-49be-b550-ffac9e4ee192`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:56,230::task::595::TaskManager.Task::(_updateState) Task=`c58349f1-1039-49be-b550-ffac9e4ee192`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:56,231::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:56,231::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:56,231::task::990::TaskManager.Task::(_decref) Task=`c58349f1-1039-49be-b550-ffac9e4ee192`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:57,236::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:57,237::task::595::TaskManager.Task::(_updateState) Task=`c9b5379d-28ea-4cbd-95b5-0f0c7b432889`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:57,237::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:57,238::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:57,238::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:57,238::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:57,239::task::1185::TaskManager.Task::(prepare) Task=`c9b5379d-28ea-4cbd-95b5-0f0c7b432889`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:57,239::task::595::TaskManager.Task::(_updateState) Task=`c9b5379d-28ea-4cbd-95b5-0f0c7b432889`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:57,239::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:57,239::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:57,240::task::990::TaskManager.Task::(_decref) Task=`c9b5379d-28ea-4cbd-95b5-0f0c7b432889`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:58,244::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:58,245::task::595::TaskManager.Task::(_updateState) Task=`50f6289c-5eba-4b2e-a1cc-1fd43b58479c`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:58,246::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:58,246::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:58,246::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:58,247::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:58,247::task::1185::TaskManager.Task::(prepare) Task=`50f6289c-5eba-4b2e-a1cc-1fd43b58479c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:58,247::task::595::TaskManager.Task::(_updateState) Task=`50f6289c-5eba-4b2e-a1cc-1fd43b58479c`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:58,248::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:58,248::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:58,249::task::990::TaskManager.Task::(_decref) Task=`50f6289c-5eba-4b2e-a1cc-1fd43b58479c`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:18:59,253::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:18:59,254::task::595::TaskManager.Task::(_updateState) Task=`b436693c-4ee6-4646-be23-ca19c46fab61`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:18:59,254::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:18:59,254::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:18:59,255::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:18:59,255::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:59,255::task::1185::TaskManager.Task::(prepare) Task=`b436693c-4ee6-4646-be23-ca19c46fab61`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:18:59,256::task::595::TaskManager.Task::(_updateState) Task=`b436693c-4ee6-4646-be23-ca19c46fab61`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:18:59,256::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:18:59,256::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:18:59,257::task::990::TaskManager.Task::(_decref) Task=`b436693c-4ee6-4646-be23-ca19c46fab61`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:00,267::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:00,268::task::595::TaskManager.Task::(_updateState) Task=`ca7208d2-4c79-4a81-b32b-8c09d5ec54e9`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:00,270::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:00,272::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:00,273::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:00,274::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:00,276::task::1185::TaskManager.Task::(prepare) Task=`ca7208d2-4c79-4a81-b32b-8c09d5ec54e9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:00,277::task::595::TaskManager.Task::(_updateState) Task=`ca7208d2-4c79-4a81-b32b-8c09d5ec54e9`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:00,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:00,279::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:00,279::task::990::TaskManager.Task::(_decref) Task=`ca7208d2-4c79-4a81-b32b-8c09d5ec54e9`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:01,285::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:01,287::task::595::TaskManager.Task::(_updateState) Task=`93e1561b-0f76-44d7-8888-9b43a41f50cf`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:01,287::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:01,287::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:01,288::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:01,289::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:01,289::task::1185::TaskManager.Task::(prepare) Task=`93e1561b-0f76-44d7-8888-9b43a41f50cf`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:01,290::task::595::TaskManager.Task::(_updateState) Task=`93e1561b-0f76-44d7-8888-9b43a41f50cf`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:01,291::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:01,291::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:01,292::task::990::TaskManager.Task::(_decref) Task=`93e1561b-0f76-44d7-8888-9b43a41f50cf`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:02,297::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:02,299::task::595::TaskManager.Task::(_updateState) Task=`a27312bc-86ca-4f02-9d00-f927af2bd15a`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:02,299::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:02,300::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:02,301::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:02,301::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:02,301::task::1185::TaskManager.Task::(prepare) Task=`a27312bc-86ca-4f02-9d00-f927af2bd15a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:02,302::task::595::TaskManager.Task::(_updateState) Task=`a27312bc-86ca-4f02-9d00-f927af2bd15a`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:02,304::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:02,304::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:02,305::task::990::TaskManager.Task::(_decref) Task=`a27312bc-86ca-4f02-9d00-f927af2bd15a`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:03,311::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:03,324::task::595::TaskManager.Task::(_updateState) Task=`9439906d-c076-4ae5-a489-dcf71502341d`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:03,324::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:03,324::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:03,325::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:03,325::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:03,325::task::1185::TaskManager.Task::(prepare) Task=`9439906d-c076-4ae5-a489-dcf71502341d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:03,326::task::595::TaskManager.Task::(_updateState) Task=`9439906d-c076-4ae5-a489-dcf71502341d`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:03,326::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:03,327::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:03,327::task::990::TaskManager.Task::(_decref) Task=`9439906d-c076-4ae5-a489-dcf71502341d`::ref 0 aborting False >Thread-73::DEBUG::2014-04-11 22:19:04,054::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-83::DEBUG::2014-04-11 22:19:04,199::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 21619c8e-99ea-4813-be02-d708971e5393 >Thread-28::DEBUG::2014-04-11 22:19:04,350::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:04,352::task::595::TaskManager.Task::(_updateState) Task=`621eff95-3b6f-48d8-9087-b1267d4a46e0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:04,352::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:04,353::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:04,353::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:04,354::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:04,354::task::1185::TaskManager.Task::(prepare) Task=`621eff95-3b6f-48d8-9087-b1267d4a46e0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:04,355::task::595::TaskManager.Task::(_updateState) Task=`621eff95-3b6f-48d8-9087-b1267d4a46e0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:04,355::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:04,356::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:04,357::task::990::TaskManager.Task::(_decref) Task=`621eff95-3b6f-48d8-9087-b1267d4a46e0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:05,368::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:05,373::task::595::TaskManager.Task::(_updateState) Task=`ecb3b498-b718-4602-a697-d49be660d3f0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:05,376::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:05,377::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:05,377::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:05,377::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:05,378::task::1185::TaskManager.Task::(prepare) Task=`ecb3b498-b718-4602-a697-d49be660d3f0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:05,378::task::595::TaskManager.Task::(_updateState) Task=`ecb3b498-b718-4602-a697-d49be660d3f0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:05,381::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:05,382::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:05,384::task::990::TaskManager.Task::(_decref) Task=`ecb3b498-b718-4602-a697-d49be660d3f0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:06,389::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:06,390::task::595::TaskManager.Task::(_updateState) Task=`bea64871-97a9-4afd-9def-05d8f7416746`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:06,391::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:06,391::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:06,392::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:06,392::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:06,393::task::1185::TaskManager.Task::(prepare) Task=`bea64871-97a9-4afd-9def-05d8f7416746`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:06,393::task::595::TaskManager.Task::(_updateState) Task=`bea64871-97a9-4afd-9def-05d8f7416746`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:06,393::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:06,394::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:06,394::task::990::TaskManager.Task::(_decref) Task=`bea64871-97a9-4afd-9def-05d8f7416746`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:07,399::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:07,400::task::595::TaskManager.Task::(_updateState) Task=`5f085ca1-6348-4cb6-a429-7068ad594188`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:07,400::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:07,400::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:07,401::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:07,401::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:07,402::task::1185::TaskManager.Task::(prepare) Task=`5f085ca1-6348-4cb6-a429-7068ad594188`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:07,402::task::595::TaskManager.Task::(_updateState) Task=`5f085ca1-6348-4cb6-a429-7068ad594188`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:07,402::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:07,403::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:07,404::task::990::TaskManager.Task::(_decref) Task=`5f085ca1-6348-4cb6-a429-7068ad594188`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:08,408::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:08,421::task::595::TaskManager.Task::(_updateState) Task=`6236e55b-343a-4248-96d6-8e9bbf8fa376`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:08,422::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:08,422::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:08,423::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:08,423::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:08,424::task::1185::TaskManager.Task::(prepare) Task=`6236e55b-343a-4248-96d6-8e9bbf8fa376`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:08,424::task::595::TaskManager.Task::(_updateState) Task=`6236e55b-343a-4248-96d6-8e9bbf8fa376`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:08,425::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:08,425::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:08,426::task::990::TaskManager.Task::(_decref) Task=`6236e55b-343a-4248-96d6-8e9bbf8fa376`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:09,431::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:09,433::task::595::TaskManager.Task::(_updateState) Task=`c76847a8-f92a-4ffa-8794-7e17098aff0c`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:09,433::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:09,434::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:09,435::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:09,436::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:09,436::task::1185::TaskManager.Task::(prepare) Task=`c76847a8-f92a-4ffa-8794-7e17098aff0c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:09,437::task::595::TaskManager.Task::(_updateState) Task=`c76847a8-f92a-4ffa-8794-7e17098aff0c`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:09,437::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:09,438::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:09,439::task::990::TaskManager.Task::(_decref) Task=`c76847a8-f92a-4ffa-8794-7e17098aff0c`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:10,489::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:10,540::task::595::TaskManager.Task::(_updateState) Task=`d642f430-ed1e-4753-b2aa-a9094dc32dec`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:10,582::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:10,613::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:10,644::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:10,676::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:10,708::task::1185::TaskManager.Task::(prepare) Task=`d642f430-ed1e-4753-b2aa-a9094dc32dec`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:10,740::task::595::TaskManager.Task::(_updateState) Task=`d642f430-ed1e-4753-b2aa-a9094dc32dec`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:10,773::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:10,804::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:10,842::task::990::TaskManager.Task::(_decref) Task=`d642f430-ed1e-4753-b2aa-a9094dc32dec`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:11,956::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:11,964::task::595::TaskManager.Task::(_updateState) Task=`dea83ab0-7414-4204-a645-381143427314`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:11,971::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:11,978::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:11,985::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:11,995::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:12,001::task::1185::TaskManager.Task::(prepare) Task=`dea83ab0-7414-4204-a645-381143427314`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:12,004::task::595::TaskManager.Task::(_updateState) Task=`dea83ab0-7414-4204-a645-381143427314`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:12,012::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:12,028::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:12,037::task::990::TaskManager.Task::(_decref) Task=`dea83ab0-7414-4204-a645-381143427314`::ref 0 aborting False >Thread-284::WARNING::2014-04-11 22:19:12,188::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-285::WARNING::2014-04-11 22:19:12,197::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-284::DEBUG::2014-04-11 22:19:12,264::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-286::WARNING::2014-04-11 22:19:12,252::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-287::WARNING::2014-04-11 22:19:12,267::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-286::DEBUG::2014-04-11 22:19:12,326::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-288::WARNING::2014-04-11 22:19:12,270::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-287::DEBUG::2014-04-11 22:19:12,345::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-289::WARNING::2014-04-11 22:19:12,276::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-288::DEBUG::2014-04-11 22:19:12,369::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-290::WARNING::2014-04-11 22:19:12,282::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-291::WARNING::2014-04-11 22:19:12,285::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-290::DEBUG::2014-04-11 22:19:12,414::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-295::WARNING::2014-04-11 22:19:12,391::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-291::DEBUG::2014-04-11 22:19:12,432::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-72::ERROR::2014-04-11 22:19:12,443::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-296::WARNING::2014-04-11 22:19:12,421::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-295::DEBUG::2014-04-11 22:19:12,452::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-297::WARNING::2014-04-11 22:19:12,427::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-81::ERROR::2014-04-11 22:19:12,461::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-72::ERROR::2014-04-11 22:19:12,469::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::ERROR::2014-04-11 22:19:12,471::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-298::WARNING::2014-04-11 22:19:12,431::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-296::DEBUG::2014-04-11 22:19:12,478::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-297::DEBUG::2014-04-11 22:19:12,489::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-81::ERROR::2014-04-11 22:19:12,496::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-301::WARNING::2014-04-11 22:19:12,499::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >9662096e-137a-47a2-8c7f-a26a6b76bcb5::ERROR::2014-04-11 22:19:12,512::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-298::DEBUG::2014-04-11 22:19:12,521::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-302::WARNING::2014-04-11 22:19:12,531::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-301::DEBUG::2014-04-11 22:19:12,545::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-303::WARNING::2014-04-11 22:19:12,532::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:19:12,552::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-302::DEBUG::2014-04-11 22:19:12,556::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-74::ERROR::2014-04-11 22:19:12,556::sdc::143::Storage.StorageDomainCache::(_findDomain) domain b7663d70-e658-41fa-b9f0-8da83c9eddce not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-304::WARNING::2014-04-11 22:19:12,537::fileSD::639::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 628, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction > rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll > raise Timeout() >Timeout >Thread-303::DEBUG::2014-04-11 22:19:12,566::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-74::ERROR::2014-04-11 22:19:12,574::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain b7663d70-e658-41fa-b9f0-8da83c9eddce monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-304::DEBUG::2014-04-11 22:19:12,578::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: '' >Thread-82::ERROR::2014-04-11 22:19:12,587::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:19:12,590::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8' (cwd None) >Thread-82::ERROR::2014-04-11 22:19:12,590::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:19:12,917::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n'; <rc> = 5 >9662096e-137a-47a2-8c7f-a26a6b76bcb5::WARNING::2014-04-11 22:19:12,922::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8'] >9662096e-137a-47a2-8c7f-a26a6b76bcb5::DEBUG::2014-04-11 22:19:12,922::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:19:13,067::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:13,070::task::595::TaskManager.Task::(_updateState) Task=`1b4d43fc-5e71-440e-b87d-5a018cfe1219`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:13,071::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:13,071::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:13,072::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:13,073::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:13,074::task::1185::TaskManager.Task::(prepare) Task=`1b4d43fc-5e71-440e-b87d-5a018cfe1219`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:13,074::task::595::TaskManager.Task::(_updateState) Task=`1b4d43fc-5e71-440e-b87d-5a018cfe1219`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:13,075::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:13,075::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:13,075::task::990::TaskManager.Task::(_decref) Task=`1b4d43fc-5e71-440e-b87d-5a018cfe1219`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:14,080::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:14,082::task::595::TaskManager.Task::(_updateState) Task=`473667e2-7567-4d58-bbbb-b687206888c9`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:14,082::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:14,083::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:14,083::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:14,083::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:14,084::task::1185::TaskManager.Task::(prepare) Task=`473667e2-7567-4d58-bbbb-b687206888c9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:14,084::task::595::TaskManager.Task::(_updateState) Task=`473667e2-7567-4d58-bbbb-b687206888c9`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:14,084::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:14,085::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:14,085::task::990::TaskManager.Task::(_decref) Task=`473667e2-7567-4d58-bbbb-b687206888c9`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:15,090::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:15,092::task::595::TaskManager.Task::(_updateState) Task=`2ea8fd68-6e93-4ef3-83f6-063d4bcddfc0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:15,093::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:15,093::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:15,094::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:15,095::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:15,095::task::1185::TaskManager.Task::(prepare) Task=`2ea8fd68-6e93-4ef3-83f6-063d4bcddfc0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:15,095::task::595::TaskManager.Task::(_updateState) Task=`2ea8fd68-6e93-4ef3-83f6-063d4bcddfc0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:15,096::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:15,096::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:15,096::task::990::TaskManager.Task::(_decref) Task=`2ea8fd68-6e93-4ef3-83f6-063d4bcddfc0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:16,101::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:16,102::task::595::TaskManager.Task::(_updateState) Task=`1811c807-83b8-4971-8d09-5d822bec2d16`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:16,102::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:16,102::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:16,103::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:16,103::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:16,103::task::1185::TaskManager.Task::(prepare) Task=`1811c807-83b8-4971-8d09-5d822bec2d16`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:16,104::task::595::TaskManager.Task::(_updateState) Task=`1811c807-83b8-4971-8d09-5d822bec2d16`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:16,104::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:16,104::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:16,105::task::990::TaskManager.Task::(_decref) Task=`1811c807-83b8-4971-8d09-5d822bec2d16`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:17,110::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:17,112::task::595::TaskManager.Task::(_updateState) Task=`799e7cfd-5f02-4d76-a1e4-75b9e84dbbb2`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:17,113::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:17,113::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:17,113::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:17,114::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:17,114::task::1185::TaskManager.Task::(prepare) Task=`799e7cfd-5f02-4d76-a1e4-75b9e84dbbb2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:17,114::task::595::TaskManager.Task::(_updateState) Task=`799e7cfd-5f02-4d76-a1e4-75b9e84dbbb2`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:17,115::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:17,115::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:17,116::task::990::TaskManager.Task::(_decref) Task=`799e7cfd-5f02-4d76-a1e4-75b9e84dbbb2`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:18,120::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:18,121::task::595::TaskManager.Task::(_updateState) Task=`caf19566-078e-4b56-875b-957915ebe743`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:18,121::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:18,122::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:18,122::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:18,122::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:18,122::task::1185::TaskManager.Task::(prepare) Task=`caf19566-078e-4b56-875b-957915ebe743`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:18,123::task::595::TaskManager.Task::(_updateState) Task=`caf19566-078e-4b56-875b-957915ebe743`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:18,123::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:18,123::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:18,124::task::990::TaskManager.Task::(_decref) Task=`caf19566-078e-4b56-875b-957915ebe743`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:19,128::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:19,144::task::595::TaskManager.Task::(_updateState) Task=`f999d5db-b0b4-4db2-8c5d-4d31d2dd3b71`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:19,145::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:19,145::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:19,145::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:19,146::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:19,146::task::1185::TaskManager.Task::(prepare) Task=`f999d5db-b0b4-4db2-8c5d-4d31d2dd3b71`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:19,146::task::595::TaskManager.Task::(_updateState) Task=`f999d5db-b0b4-4db2-8c5d-4d31d2dd3b71`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:19,147::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:19,147::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:19,147::task::990::TaskManager.Task::(_decref) Task=`f999d5db-b0b4-4db2-8c5d-4d31d2dd3b71`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:20,152::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:20,153::task::595::TaskManager.Task::(_updateState) Task=`46e73d63-e7b8-4c7d-ad0d-876fe3ddf4fd`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:20,154::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:20,154::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:20,154::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:20,155::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:20,155::task::1185::TaskManager.Task::(prepare) Task=`46e73d63-e7b8-4c7d-ad0d-876fe3ddf4fd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:20,155::task::595::TaskManager.Task::(_updateState) Task=`46e73d63-e7b8-4c7d-ad0d-876fe3ddf4fd`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:20,156::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:20,156::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:20,156::task::990::TaskManager.Task::(_decref) Task=`46e73d63-e7b8-4c7d-ad0d-876fe3ddf4fd`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:21,161::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:21,161::task::595::TaskManager.Task::(_updateState) Task=`9bcfe900-db3f-498d-b390-459d24f37bfb`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:21,162::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:21,162::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:21,162::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:21,163::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:21,163::task::1185::TaskManager.Task::(prepare) Task=`9bcfe900-db3f-498d-b390-459d24f37bfb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:21,163::task::595::TaskManager.Task::(_updateState) Task=`9bcfe900-db3f-498d-b390-459d24f37bfb`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:21,164::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:21,164::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:21,164::task::990::TaskManager.Task::(_decref) Task=`9bcfe900-db3f-498d-b390-459d24f37bfb`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:22,168::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:22,169::task::595::TaskManager.Task::(_updateState) Task=`9ce527db-2332-443e-ad78-0aae11470a93`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:22,169::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:22,169::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:22,170::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:22,170::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:22,171::task::1185::TaskManager.Task::(prepare) Task=`9ce527db-2332-443e-ad78-0aae11470a93`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:22,171::task::595::TaskManager.Task::(_updateState) Task=`9ce527db-2332-443e-ad78-0aae11470a93`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:22,171::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:22,172::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:22,172::task::990::TaskManager.Task::(_decref) Task=`9ce527db-2332-443e-ad78-0aae11470a93`::ref 0 aborting False >Thread-72::DEBUG::2014-04-11 22:19:22,511::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-81::DEBUG::2014-04-11 22:19:22,540::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-81::ERROR::2014-04-11 22:19:22,540::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-81::ERROR::2014-04-11 22:19:22,541::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-81::DEBUG::2014-04-11 22:19:22,541::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-81::DEBUG::2014-04-11 22:19:22,542::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-74::DEBUG::2014-04-11 22:19:22,585::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-74::ERROR::2014-04-11 22:19:22,585::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-74::ERROR::2014-04-11 22:19:22,586::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-74::DEBUG::2014-04-11 22:19:22,586::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-74::DEBUG::2014-04-11 22:19:22,587::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name b7663d70-e658-41fa-b9f0-8da83c9eddce' (cwd None) >Thread-82::DEBUG::2014-04-11 22:19:22,609::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-82::ERROR::2014-04-11 22:19:22,610::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-82::ERROR::2014-04-11 22:19:22,611::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-82::DEBUG::2014-04-11 22:19:22,611::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-82::DEBUG::2014-04-11 22:19:22,613::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-81::DEBUG::2014-04-11 22:19:22,853::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-81::WARNING::2014-04-11 22:19:22,856::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-74::DEBUG::2014-04-11 22:19:22,904::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n'; <rc> = 5 >Thread-82::DEBUG::2014-04-11 22:19:22,983::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-74::WARNING::2014-04-11 22:19:22,988::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce'] >Thread-82::WARNING::2014-04-11 22:19:22,999::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-82::DEBUG::2014-04-11 22:19:23,005::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28::DEBUG::2014-04-11 22:19:23,256::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:23,264::task::595::TaskManager.Task::(_updateState) Task=`15267231-eca5-4bf4-87e4-ad6040bd6b79`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:23,264::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:23,264::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:23,265::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:23,265::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:23,266::task::1185::TaskManager.Task::(prepare) Task=`15267231-eca5-4bf4-87e4-ad6040bd6b79`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:23,266::task::595::TaskManager.Task::(_updateState) Task=`15267231-eca5-4bf4-87e4-ad6040bd6b79`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:23,266::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:23,267::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:23,268::task::990::TaskManager.Task::(_decref) Task=`15267231-eca5-4bf4-87e4-ad6040bd6b79`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:24,272::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:24,273::task::595::TaskManager.Task::(_updateState) Task=`7177d31b-3931-48b6-8262-f4a6e81a492b`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:24,274::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:24,274::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:24,274::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:24,275::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:24,275::task::1185::TaskManager.Task::(prepare) Task=`7177d31b-3931-48b6-8262-f4a6e81a492b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:24,275::task::595::TaskManager.Task::(_updateState) Task=`7177d31b-3931-48b6-8262-f4a6e81a492b`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:24,276::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:24,276::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:24,276::task::990::TaskManager.Task::(_decref) Task=`7177d31b-3931-48b6-8262-f4a6e81a492b`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:25,282::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:25,284::task::595::TaskManager.Task::(_updateState) Task=`e6ca6b07-1d05-464a-adb8-fa158b6f0ad0`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:25,284::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:25,285::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:25,285::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:25,286::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:25,286::task::1185::TaskManager.Task::(prepare) Task=`e6ca6b07-1d05-464a-adb8-fa158b6f0ad0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:25,286::task::595::TaskManager.Task::(_updateState) Task=`e6ca6b07-1d05-464a-adb8-fa158b6f0ad0`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:25,287::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:25,287::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:25,287::task::990::TaskManager.Task::(_decref) Task=`e6ca6b07-1d05-464a-adb8-fa158b6f0ad0`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:26,293::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:26,293::task::595::TaskManager.Task::(_updateState) Task=`add95d90-907c-40e9-a455-b54cc69ca8a7`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:26,295::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:26,295::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:26,296::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:26,297::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:26,297::task::1185::TaskManager.Task::(prepare) Task=`add95d90-907c-40e9-a455-b54cc69ca8a7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:26,298::task::595::TaskManager.Task::(_updateState) Task=`add95d90-907c-40e9-a455-b54cc69ca8a7`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:26,298::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:26,298::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:26,299::task::990::TaskManager.Task::(_decref) Task=`add95d90-907c-40e9-a455-b54cc69ca8a7`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:27,303::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:27,304::task::595::TaskManager.Task::(_updateState) Task=`b6814913-1c85-46f5-b2a5-fc08e01bf832`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:27,305::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:27,305::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:27,305::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:27,306::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:27,306::task::1185::TaskManager.Task::(prepare) Task=`b6814913-1c85-46f5-b2a5-fc08e01bf832`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:27,307::task::595::TaskManager.Task::(_updateState) Task=`b6814913-1c85-46f5-b2a5-fc08e01bf832`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:27,307::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:27,307::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:27,308::task::990::TaskManager.Task::(_decref) Task=`b6814913-1c85-46f5-b2a5-fc08e01bf832`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:28,312::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:28,313::task::595::TaskManager.Task::(_updateState) Task=`aba5a048-3300-4ab3-95ed-a02805f09d5a`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:28,313::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:28,313::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:28,314::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:28,314::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:28,314::task::1185::TaskManager.Task::(prepare) Task=`aba5a048-3300-4ab3-95ed-a02805f09d5a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:28,314::task::595::TaskManager.Task::(_updateState) Task=`aba5a048-3300-4ab3-95ed-a02805f09d5a`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:28,315::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:28,315::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:28,315::task::990::TaskManager.Task::(_decref) Task=`aba5a048-3300-4ab3-95ed-a02805f09d5a`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:29,319::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:29,320::task::595::TaskManager.Task::(_updateState) Task=`6c9b0c83-377e-482f-bd14-546ee9f72c39`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:29,321::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:29,321::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:29,321::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:29,321::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:29,322::task::1185::TaskManager.Task::(prepare) Task=`6c9b0c83-377e-482f-bd14-546ee9f72c39`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:29,322::task::595::TaskManager.Task::(_updateState) Task=`6c9b0c83-377e-482f-bd14-546ee9f72c39`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:29,322::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:29,322::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:29,323::task::990::TaskManager.Task::(_decref) Task=`6c9b0c83-377e-482f-bd14-546ee9f72c39`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:30,328::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:30,329::task::595::TaskManager.Task::(_updateState) Task=`0d87e83f-9cee-446c-bb09-f1a030e2f40d`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:30,330::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:30,330::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:30,331::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:30,332::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:30,332::task::1185::TaskManager.Task::(prepare) Task=`0d87e83f-9cee-446c-bb09-f1a030e2f40d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:30,332::task::595::TaskManager.Task::(_updateState) Task=`0d87e83f-9cee-446c-bb09-f1a030e2f40d`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:30,333::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:30,334::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:30,335::task::990::TaskManager.Task::(_decref) Task=`0d87e83f-9cee-446c-bb09-f1a030e2f40d`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:31,341::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:31,342::task::595::TaskManager.Task::(_updateState) Task=`7261bcd7-3354-4dbe-8386-56c49a13461a`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:31,343::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:31,343::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:31,343::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:31,344::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:31,345::task::1185::TaskManager.Task::(prepare) Task=`7261bcd7-3354-4dbe-8386-56c49a13461a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:31,345::task::595::TaskManager.Task::(_updateState) Task=`7261bcd7-3354-4dbe-8386-56c49a13461a`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:31,346::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:31,346::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:31,347::task::990::TaskManager.Task::(_decref) Task=`7261bcd7-3354-4dbe-8386-56c49a13461a`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:32,351::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:32,352::task::595::TaskManager.Task::(_updateState) Task=`e2eecfc3-daf6-499e-8d21-cadff077adcb`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:32,353::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:32,353::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:32,354::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:32,354::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:32,355::task::1185::TaskManager.Task::(prepare) Task=`e2eecfc3-daf6-499e-8d21-cadff077adcb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:32,355::task::595::TaskManager.Task::(_updateState) Task=`e2eecfc3-daf6-499e-8d21-cadff077adcb`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:32,356::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:32,357::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:32,357::task::990::TaskManager.Task::(_decref) Task=`e2eecfc3-daf6-499e-8d21-cadff077adcb`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:33,362::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:33,362::task::595::TaskManager.Task::(_updateState) Task=`00fb333b-4c18-455e-85aa-2578acb9e0fa`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:33,363::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:33,364::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:33,364::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:33,364::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:33,365::task::1185::TaskManager.Task::(prepare) Task=`00fb333b-4c18-455e-85aa-2578acb9e0fa`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:33,365::task::595::TaskManager.Task::(_updateState) Task=`00fb333b-4c18-455e-85aa-2578acb9e0fa`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:33,365::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:33,365::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:33,366::task::990::TaskManager.Task::(_decref) Task=`00fb333b-4c18-455e-85aa-2578acb9e0fa`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:34,370::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:34,371::task::595::TaskManager.Task::(_updateState) Task=`6b17fed8-3f4b-4368-b881-9403b047d927`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:34,371::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:34,371::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:34,372::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:34,372::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:34,372::task::1185::TaskManager.Task::(prepare) Task=`6b17fed8-3f4b-4368-b881-9403b047d927`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:34,373::task::595::TaskManager.Task::(_updateState) Task=`6b17fed8-3f4b-4368-b881-9403b047d927`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:34,373::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:34,373::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:34,373::task::990::TaskManager.Task::(_decref) Task=`6b17fed8-3f4b-4368-b881-9403b047d927`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:35,379::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:35,380::task::595::TaskManager.Task::(_updateState) Task=`3a766c95-e5c7-4ce4-9161-a2276b9eb1e6`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:35,382::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:35,382::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:35,382::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:35,383::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:35,383::task::1185::TaskManager.Task::(prepare) Task=`3a766c95-e5c7-4ce4-9161-a2276b9eb1e6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:35,385::task::595::TaskManager.Task::(_updateState) Task=`3a766c95-e5c7-4ce4-9161-a2276b9eb1e6`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:35,386::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:35,386::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:35,387::task::990::TaskManager.Task::(_decref) Task=`3a766c95-e5c7-4ce4-9161-a2276b9eb1e6`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:36,392::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:36,393::task::595::TaskManager.Task::(_updateState) Task=`504a5a5a-93a0-426d-ba3f-472111029615`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:36,393::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:36,393::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:36,394::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:36,394::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:36,394::task::1185::TaskManager.Task::(prepare) Task=`504a5a5a-93a0-426d-ba3f-472111029615`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:36,395::task::595::TaskManager.Task::(_updateState) Task=`504a5a5a-93a0-426d-ba3f-472111029615`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:36,395::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:36,395::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:36,395::task::990::TaskManager.Task::(_decref) Task=`504a5a5a-93a0-426d-ba3f-472111029615`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:37,400::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:37,401::task::595::TaskManager.Task::(_updateState) Task=`f7874fcc-5d27-465e-ade6-0404a41d1b4b`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:37,401::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:37,402::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:37,402::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:37,402::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:37,402::task::1185::TaskManager.Task::(prepare) Task=`f7874fcc-5d27-465e-ade6-0404a41d1b4b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:37,403::task::595::TaskManager.Task::(_updateState) Task=`f7874fcc-5d27-465e-ade6-0404a41d1b4b`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:37,403::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:37,403::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:37,404::task::990::TaskManager.Task::(_decref) Task=`f7874fcc-5d27-465e-ade6-0404a41d1b4b`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:38,408::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:38,408::task::595::TaskManager.Task::(_updateState) Task=`d279ba4f-bb4e-45df-ba3c-bae03ca7fa9b`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:38,409::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:38,409::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:38,409::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:38,410::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:38,410::task::1185::TaskManager.Task::(prepare) Task=`d279ba4f-bb4e-45df-ba3c-bae03ca7fa9b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:38,410::task::595::TaskManager.Task::(_updateState) Task=`d279ba4f-bb4e-45df-ba3c-bae03ca7fa9b`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:38,411::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:38,411::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:38,411::task::990::TaskManager.Task::(_decref) Task=`d279ba4f-bb4e-45df-ba3c-bae03ca7fa9b`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:39,415::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:39,416::task::595::TaskManager.Task::(_updateState) Task=`ee7f9449-acec-49c8-9d4c-b0c751e23c84`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:39,417::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:39,417::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:39,417::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:39,418::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:39,418::task::1185::TaskManager.Task::(prepare) Task=`ee7f9449-acec-49c8-9d4c-b0c751e23c84`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:39,418::task::595::TaskManager.Task::(_updateState) Task=`ee7f9449-acec-49c8-9d4c-b0c751e23c84`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:39,418::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:39,419::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:39,419::task::990::TaskManager.Task::(_decref) Task=`ee7f9449-acec-49c8-9d4c-b0c751e23c84`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:40,433::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:40,434::task::595::TaskManager.Task::(_updateState) Task=`e9859e0e-53e3-440f-9aca-3da5152b3255`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:40,435::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:40,435::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:40,435::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:40,436::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:40,436::task::1185::TaskManager.Task::(prepare) Task=`e9859e0e-53e3-440f-9aca-3da5152b3255`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:40,437::task::595::TaskManager.Task::(_updateState) Task=`e9859e0e-53e3-440f-9aca-3da5152b3255`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:40,437::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:40,438::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:40,438::task::990::TaskManager.Task::(_decref) Task=`e9859e0e-53e3-440f-9aca-3da5152b3255`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:41,444::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:41,445::task::595::TaskManager.Task::(_updateState) Task=`39f37e0a-7d01-4950-8992-8701847b78ef`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:41,445::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:41,445::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:41,447::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:41,448::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:41,448::task::1185::TaskManager.Task::(prepare) Task=`39f37e0a-7d01-4950-8992-8701847b78ef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:41,448::task::595::TaskManager.Task::(_updateState) Task=`39f37e0a-7d01-4950-8992-8701847b78ef`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:41,449::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:41,449::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:41,450::task::990::TaskManager.Task::(_decref) Task=`39f37e0a-7d01-4950-8992-8701847b78ef`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:42,454::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [19caef6e] >Thread-28::DEBUG::2014-04-11 22:19:42,455::task::595::TaskManager.Task::(_updateState) Task=`e3f1f0fd-5db8-4cb4-804a-f6a77897b340`::moving from state init -> state preparing >Thread-28::INFO::2014-04-11 22:19:42,456::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9662096e-137a-47a2-8c7f-a26a6b76bcb5', spUUID=None, options=None) >Thread-28::DEBUG::2014-04-11 22:19:42,456::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9662096e-137a-47a2-8c7f-a26a6b76bcb5 >Thread-28::DEBUG::2014-04-11 22:19:42,456::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'} >Thread-28::INFO::2014-04-11 22:19:42,457::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:42,457::task::1185::TaskManager.Task::(prepare) Task=`e3f1f0fd-5db8-4cb4-804a-f6a77897b340`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9662096e-137a-47a2-8c7f-a26a6b76bcb5'}} >Thread-28::DEBUG::2014-04-11 22:19:42,457::task::595::TaskManager.Task::(_updateState) Task=`e3f1f0fd-5db8-4cb4-804a-f6a77897b340`::moving from state preparing -> state finished >Thread-28::DEBUG::2014-04-11 22:19:42,458::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28::DEBUG::2014-04-11 22:19:42,458::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28::DEBUG::2014-04-11 22:19:42,458::task::990::TaskManager.Task::(_decref) Task=`e3f1f0fd-5db8-4cb4-804a-f6a77897b340`::ref 0 aborting False >Thread-28::DEBUG::2014-04-11 22:19:43,463::BindingXMLRPC::251::vds::(wrapper) clientMainThread::INFO::2014-04-11 22:22:43,596::vdsm::119::vds::(run) (PID: 4571) I am the actual vdsm 4.14.6-12.git49cacf0.el6 ashtivh01.suprtekstic.com (2.6.32-431.11.2.el6.x86_64) >MainThread::DEBUG::2014-04-11 22:22:43,605::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2014-04-11 22:22:43,606::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 >MainThread::WARNING::2014-04-11 22:22:43,612::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists >MainThread::DEBUG::2014-04-11 22:22:45,249::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >MainThread::DEBUG::2014-04-11 22:22:45,286::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving >MainThread::DEBUG::2014-04-11 22:22:45,286::hsm::424::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2014-04-11 22:22:45,554::hsm::424::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2014-04-11 22:22:45,555::hsm::448::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >MainThread::DEBUG::2014-04-11 22:22:45,561::hsm::480::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >MainThread::DEBUG::2014-04-11 22:22:45,562::hsm::481::Storage.HSM::(__cleanStorageRepository) Mount list: [] >MainThread::DEBUG::2014-04-11 22:22:45,562::hsm::483::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >MainThread::DEBUG::2014-04-11 22:22:45,596::hsm::526::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >storageRefresh::DEBUG::2014-04-11 22:22:45,598::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >MainThread::INFO::2014-04-11 22:22:45,601::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >storageRefresh::DEBUG::2014-04-11 22:22:45,601::misc::764::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-04-11 22:22:45,803::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >storageRefresh::DEBUG::2014-04-11 22:22:45,803::misc::764::SamplingMethod::(__call__) Got in to sampling method >storageRefresh::DEBUG::2014-04-11 22:22:45,808::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >storageRefresh::DEBUG::2014-04-11 22:22:45,809::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:22:45,892::misc::772::SamplingMethod::(__call__) Returning last result >storageRefresh::DEBUG::2014-04-11 22:22:45,893::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >MainThread::DEBUG::2014-04-11 22:22:45,968::task::595::TaskManager.Task::(_updateState) Task=`4b7ade79-a63d-4da2-b641-956821a6d085`::moving from state init -> state preparing >MainThread::INFO::2014-04-11 22:22:45,968::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x2b26290>>) >MainThread::INFO::2014-04-11 22:22:45,969::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None >MainThread::DEBUG::2014-04-11 22:22:45,969::task::1185::TaskManager.Task::(prepare) Task=`4b7ade79-a63d-4da2-b641-956821a6d085`::finished: None >MainThread::DEBUG::2014-04-11 22:22:45,969::task::595::TaskManager.Task::(_updateState) Task=`4b7ade79-a63d-4da2-b641-956821a6d085`::moving from state preparing -> state finished >MainThread::DEBUG::2014-04-11 22:22:45,969::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2014-04-11 22:22:45,970::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2014-04-11 22:22:45,970::task::990::TaskManager.Task::(_decref) Task=`4b7ade79-a63d-4da2-b641-956821a6d085`::ref 0 aborting False >MainThread::INFO::2014-04-11 22:22:45,986::momIF::47::MOM::(__init__) Starting up MOM >MainThread::INFO::2014-04-11 22:22:46,004::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds. >VM Channels Listener::INFO::2014-04-11 22:22:46,006::vmChannels::170::vds::(run) Starting VM channels listener thread. >clientIFinit::DEBUG::2014-04-11 22:22:46,052::libvirtconnection::145::root::(get) trying to connect libvirt >storageRefresh::DEBUG::2014-04-11 22:22:46,055::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:22:46,056::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,057::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,063::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,064::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,064::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,064::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,065::misc::772::SamplingMethod::(__call__) Returning last result >storageRefresh::DEBUG::2014-04-11 22:22:46,065::lvm::324::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,070::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >MainThread::WARNING::2014-04-11 22:22:46,081::clientIF::181::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed. >storageRefresh::DEBUG::2014-04-11 22:22:46,410::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:22:46,415::lvm::349::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,419::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,429::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:22:46,718::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:22:46,718::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,719::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:22:46,938::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:22:46,939::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >storageRefresh::DEBUG::2014-04-11 22:22:46,941::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None) >storageRefresh::DEBUG::2014-04-11 22:22:47,182::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0 >storageRefresh::DEBUG::2014-04-11 22:22:47,183::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >storageRefresh::WARNING::2014-04-11 22:22:47,184::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists >storageRefresh::DEBUG::2014-04-11 22:22:47,185::hsm::392::Storage.HSM::(storageRefresh) HSM is ready >Thread-13::DEBUG::2014-04-11 22:22:50,198::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [744e80e3] >Thread-13::DEBUG::2014-04-11 22:22:50,199::task::595::TaskManager.Task::(_updateState) Task=`114efa06-7908-4145-89a4-9139d71e4075`::moving from state init -> state preparing >Thread-13::INFO::2014-04-11 22:22:50,199::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-13::ERROR::2014-04-11 22:22:50,200::task::866::TaskManager.Task::(_setError) Task=`114efa06-7908-4145-89a4-9139d71e4075`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses > raise se.SpmStatusError() >SpmStatusError: Not SPM: () >Thread-13::DEBUG::2014-04-11 22:22:50,227::task::885::TaskManager.Task::(_run) Task=`114efa06-7908-4145-89a4-9139d71e4075`::Task._run: 114efa06-7908-4145-89a4-9139d71e4075 () {} failed - stopping task >Thread-13::DEBUG::2014-04-11 22:22:50,227::task::1211::TaskManager.Task::(stop) Task=`114efa06-7908-4145-89a4-9139d71e4075`::stopping in state preparing (force False) >Thread-13::DEBUG::2014-04-11 22:22:50,227::task::990::TaskManager.Task::(_decref) Task=`114efa06-7908-4145-89a4-9139d71e4075`::ref 1 aborting True >Thread-13::INFO::2014-04-11 22:22:50,228::task::1168::TaskManager.Task::(prepare) Task=`114efa06-7908-4145-89a4-9139d71e4075`::aborting: Task is aborted: 'Not SPM' - code 654 >Thread-13::DEBUG::2014-04-11 22:22:50,228::task::1173::TaskManager.Task::(prepare) Task=`114efa06-7908-4145-89a4-9139d71e4075`::Prepare: aborted: Not SPM >Thread-13::DEBUG::2014-04-11 22:22:50,228::task::990::TaskManager.Task::(_decref) Task=`114efa06-7908-4145-89a4-9139d71e4075`::ref 0 aborting True >Thread-13::DEBUG::2014-04-11 22:22:50,229::task::925::TaskManager.Task::(_doAbort) Task=`114efa06-7908-4145-89a4-9139d71e4075`::Task._doAbort: force False >Thread-13::DEBUG::2014-04-11 22:22:50,229::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::DEBUG::2014-04-11 22:22:50,229::task::595::TaskManager.Task::(_updateState) Task=`114efa06-7908-4145-89a4-9139d71e4075`::moving from state preparing -> state aborting >Thread-13::DEBUG::2014-04-11 22:22:50,229::task::550::TaskManager.Task::(__state_aborting) Task=`114efa06-7908-4145-89a4-9139d71e4075`::_aborting: recover policy none >Thread-13::DEBUG::2014-04-11 22:22:50,230::task::595::TaskManager.Task::(_updateState) Task=`114efa06-7908-4145-89a4-9139d71e4075`::moving from state aborting -> state failed >Thread-13::DEBUG::2014-04-11 22:22:50,230::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-13::DEBUG::2014-04-11 22:22:50,230::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-13::ERROR::2014-04-11 22:22:50,230::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}} >Thread-14::DEBUG::2014-04-11 22:53:34,883::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getCapabilities with () {} flowID [1e4e67ba] >Thread-14::DEBUG::2014-04-11 22:53:35,049::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:35,065::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:35,106::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:35,117::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:35,123::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:35,132::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:35,138::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.5' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:35,147::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:35,152::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.6' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:35,160::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:35,470::caps::458::root::(_getKeyPackages) rpm package glusterfs-rdma not found >Thread-14::DEBUG::2014-04-11 22:53:35,470::caps::458::root::(_getKeyPackages) rpm package glusterfs-fuse not found >Thread-14::DEBUG::2014-04-11 22:53:35,472::caps::458::root::(_getKeyPackages) rpm package gluster-swift not found >Thread-14::DEBUG::2014-04-11 22:53:35,479::caps::458::root::(_getKeyPackages) rpm package gluster-swift-object not found >Thread-14::DEBUG::2014-04-11 22:53:35,480::caps::458::root::(_getKeyPackages) rpm package glusterfs not found >Thread-14::DEBUG::2014-04-11 22:53:35,480::caps::458::root::(_getKeyPackages) rpm package gluster-swift-plugin not found >Thread-14::DEBUG::2014-04-11 22:53:35,490::caps::458::root::(_getKeyPackages) rpm package gluster-swift-account not found >Thread-14::DEBUG::2014-04-11 22:53:35,490::caps::458::root::(_getKeyPackages) rpm package gluster-swift-proxy not found >Thread-14::DEBUG::2014-04-11 22:53:35,491::caps::458::root::(_getKeyPackages) rpm package gluster-swift-doc not found >Thread-14::DEBUG::2014-04-11 22:53:35,491::caps::458::root::(_getKeyPackages) rpm package glusterfs-server not found >Thread-14::DEBUG::2014-04-11 22:53:35,492::caps::458::root::(_getKeyPackages) rpm package gluster-swift-container not found >Thread-14::DEBUG::2014-04-11 22:53:35,492::caps::458::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found >Thread-14::DEBUG::2014-04-11 22:53:35,519::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:93188a588127'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395791995.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '12.git49cacf0.el6', 'buildtime': 1397150575L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz', 'hooks': {'after_vm_start': {'50_promisc': {'md5': '5ea469fa06dd7cd1e52508672a7519db'}}, 'before_vm_migrate_source': {'50_sriov': {'md5': '47a59103b7c8b1439ae17dcd5ed197cd'}, '50_hostusb': {'md5': 'e50a0975acd8bafd0052b65a787c2fdd'}, '50_scratchpad': {'md5': 'be90995f959c60c5110c6a0bef67c7b8'}, '50_floppy': {'md5': 'c68fb91d44105c8b6f2d80566314027a'}}, 'after_device_destroy': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_vm_destroy': {'50_promisc': {'md5': 'dae0434ea8ca2492422a6fde5cc47a1e'}}, 'before_vm_start': {'50_sriov': {'md5': '3ebc60cd2e4eb089820102285fad7c45'}, '50_vmfex': {'md5': '9f5abb892ddb6b3daa779985d38d9f55'}, '50_qos': {'md5': 'cdf6f89d277a886691812ce1ec2fe105'}, '50_scratchpad': {'md5': '178171eb62ba95fc548e2450013f5ad5'}, '50_numa': {'md5': '5008c2826714ac5b63748780aabd2f25'}, '50_floppy': {'md5': '202fe18705a7d4c50c40c126e8f8dbe8'}, '50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}, '50_pincpu': {'md5': '0b5fb99ff0e7acb9ad534b87c02c59e3'}, '50_hugepages': {'md5': '6713d703f9d316f1664145b700cd7904'}, '50_fileinject': {'md5': '290dafd45601e010e0f496fa5b889a76'}, '50_macspoof': {'md5': '25deea559772719b90117f2e1009d823'}, '60_checkimages': {'md5': 'abd32c723a90ab2816e2163f5e02faf7'}, '10_faqemu': {'md5': 'e3cb338be6b9075a04bed0e7400ab7e2'}, '50_directlun': {'md5': 'dba5d80fdb7119ee0f4a40c26eac8448'}, '50_smbios': {'md5': '1a6bd8eddc3a86943e558bc4d3e73bd6'}, '50_vmdisk': {'md5': '67617c6e2d796388ddc6c065a97f714e'}, '50_hostusb': {'md5': '21c58568f3eae4f0d091005e0eff891a'}, '50_isolatedprivatevlan': {'md5': '0a25d7cd0b807ea5e0dfc3318ee98c91'}}, 'after_device_migrate_destination': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'after_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_vm_migrate_destination': {'50_vmfex': {'md5': 'eac2633f47fb309b4a5b7fe76531baec'}, '50_hugepages': {'md5': '6ddc484746297308b9f9acc5fcabaf10'}, '50_directlun': {'md5': '8a29ce41de079630cb58d48e280d55c2'}}, 'after_nic_hotunplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_vm_destroy': {'50_sriov': {'md5': '5890b2899d450495a62f3611fad3c526'}, '50_directlun': {'md5': '6d05c19e68a141eeb7ee5981d68c0cfa'}, '50_hostusb': {'md5': '0a37bf0a26546e3ab5e424b13653ce2b'}, '50_hugepages': {'md5': '4a980b78f8ee908bf2c61ea9bf156f4a'}, '50_scratchpad': {'md5': '316d21a669f97e83b485b45ab0b23eee'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0', 'addr': '172.16.10.1', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'bridged': False, 'interface': u'bond0', 'ipv6gateway': '::', 'gateway': '172.16.0.1'}, 'Blue': {'iface': 'Blue', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.6']}, 'Infrastructure': {'iface': 'Infrastructure', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.4']}, 'migration': {'iface': u'bond0.5', 'addr': '', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'bridged': False, 'interface': u'bond0.5', 'ipv6gateway': '::', 'gateway': ''}, 'Red': {'iface': 'Red', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.3']}}, 'bridges': {'Blue': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.6']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'Infrastructure': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.4']}, 'Red': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.3']}}, 'uuid': '4C4C4544-0059-5310-8031-B9C04F445A31', 'lastClientIface': 'bond0', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:5F', 'MASTER': 'bond0', 'DEVICE': 'em1', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:5F', 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:60', 'MASTER': 'bond0', 'DEVICE': 'em2', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:60', 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '12', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,lahf_lm,ida,arat,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:93188a588127', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '255.255.0.0', 'addr': '172.16.10.1', 'slaves': ['em1', 'em2'], 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.10.1', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond0', 'MTU': '1500', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32057', 'cpuSpeed': '1900.070', 'version_name': 'Snow Man', 'vlans': {'bond0.3': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Red', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 3, 'mtu': '1500'}, 'bond0.4': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Infrastructure', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 4, 'mtu': '1500'}, 'bond0.5': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'DEVICE': 'bond0.5', 'NM_CONTROLLED': 'no', 'VLAN': 'yes', 'DEFROUTE': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 5, 'mtu': '1500'}, 'bond0.6': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Blue', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.6', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 6, 'mtu': '1500'}}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '24', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.10.1'}} >Thread-14::DEBUG::2014-04-11 22:53:35,557::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getHardwareInfo with () {} flowID [1e4e67ba] >Thread-14::DEBUG::2014-04-11 22:53:35,769::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R420', 'systemSerialNumber': '9YS1DZ1', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0059-5310-8031-B9C04F445A31', 'systemManufacturer': 'Dell Inc.'}} >Thread-14::DEBUG::2014-04-11 22:53:39,189::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getCapabilities with () {} >Thread-14::DEBUG::2014-04-11 22:53:39,194::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,205::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,241::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,252::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,258::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,268::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,274::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.5' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,283::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,288::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.6' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,297::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,319::caps::458::root::(_getKeyPackages) rpm package glusterfs-rdma not found >Thread-14::DEBUG::2014-04-11 22:53:39,320::caps::458::root::(_getKeyPackages) rpm package glusterfs-fuse not found >Thread-14::DEBUG::2014-04-11 22:53:39,321::caps::458::root::(_getKeyPackages) rpm package gluster-swift not found >Thread-14::DEBUG::2014-04-11 22:53:39,329::caps::458::root::(_getKeyPackages) rpm package gluster-swift-object not found >Thread-14::DEBUG::2014-04-11 22:53:39,329::caps::458::root::(_getKeyPackages) rpm package glusterfs not found >Thread-14::DEBUG::2014-04-11 22:53:39,330::caps::458::root::(_getKeyPackages) rpm package gluster-swift-plugin not found >Thread-14::DEBUG::2014-04-11 22:53:39,340::caps::458::root::(_getKeyPackages) rpm package gluster-swift-account not found >Thread-14::DEBUG::2014-04-11 22:53:39,340::caps::458::root::(_getKeyPackages) rpm package gluster-swift-proxy not found >Thread-14::DEBUG::2014-04-11 22:53:39,341::caps::458::root::(_getKeyPackages) rpm package gluster-swift-doc not found >Thread-14::DEBUG::2014-04-11 22:53:39,341::caps::458::root::(_getKeyPackages) rpm package glusterfs-server not found >Thread-14::DEBUG::2014-04-11 22:53:39,342::caps::458::root::(_getKeyPackages) rpm package gluster-swift-container not found >Thread-14::DEBUG::2014-04-11 22:53:39,342::caps::458::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found >Thread-14::DEBUG::2014-04-11 22:53:39,349::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:93188a588127'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395791995.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '12.git49cacf0.el6', 'buildtime': 1397150575L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz', 'hooks': {'after_vm_start': {'50_promisc': {'md5': '5ea469fa06dd7cd1e52508672a7519db'}}, 'before_vm_migrate_source': {'50_sriov': {'md5': '47a59103b7c8b1439ae17dcd5ed197cd'}, '50_hostusb': {'md5': 'e50a0975acd8bafd0052b65a787c2fdd'}, '50_scratchpad': {'md5': 'be90995f959c60c5110c6a0bef67c7b8'}, '50_floppy': {'md5': 'c68fb91d44105c8b6f2d80566314027a'}}, 'after_device_destroy': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_vm_destroy': {'50_promisc': {'md5': 'dae0434ea8ca2492422a6fde5cc47a1e'}}, 'before_vm_start': {'50_sriov': {'md5': '3ebc60cd2e4eb089820102285fad7c45'}, '50_vmfex': {'md5': '9f5abb892ddb6b3daa779985d38d9f55'}, '50_qos': {'md5': 'cdf6f89d277a886691812ce1ec2fe105'}, '50_scratchpad': {'md5': '178171eb62ba95fc548e2450013f5ad5'}, '50_numa': {'md5': '5008c2826714ac5b63748780aabd2f25'}, '50_floppy': {'md5': '202fe18705a7d4c50c40c126e8f8dbe8'}, '50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}, '50_pincpu': {'md5': '0b5fb99ff0e7acb9ad534b87c02c59e3'}, '50_hugepages': {'md5': '6713d703f9d316f1664145b700cd7904'}, '50_fileinject': {'md5': '290dafd45601e010e0f496fa5b889a76'}, '50_macspoof': {'md5': '25deea559772719b90117f2e1009d823'}, '60_checkimages': {'md5': 'abd32c723a90ab2816e2163f5e02faf7'}, '10_faqemu': {'md5': 'e3cb338be6b9075a04bed0e7400ab7e2'}, '50_directlun': {'md5': 'dba5d80fdb7119ee0f4a40c26eac8448'}, '50_smbios': {'md5': '1a6bd8eddc3a86943e558bc4d3e73bd6'}, '50_vmdisk': {'md5': '67617c6e2d796388ddc6c065a97f714e'}, '50_hostusb': {'md5': '21c58568f3eae4f0d091005e0eff891a'}, '50_isolatedprivatevlan': {'md5': '0a25d7cd0b807ea5e0dfc3318ee98c91'}}, 'after_device_migrate_destination': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'after_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_vm_migrate_destination': {'50_vmfex': {'md5': 'eac2633f47fb309b4a5b7fe76531baec'}, '50_hugepages': {'md5': '6ddc484746297308b9f9acc5fcabaf10'}, '50_directlun': {'md5': '8a29ce41de079630cb58d48e280d55c2'}}, 'after_nic_hotunplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_vm_destroy': {'50_sriov': {'md5': '5890b2899d450495a62f3611fad3c526'}, '50_directlun': {'md5': '6d05c19e68a141eeb7ee5981d68c0cfa'}, '50_hostusb': {'md5': '0a37bf0a26546e3ab5e424b13653ce2b'}, '50_hugepages': {'md5': '4a980b78f8ee908bf2c61ea9bf156f4a'}, '50_scratchpad': {'md5': '316d21a669f97e83b485b45ab0b23eee'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0', 'addr': '172.16.10.1', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'bridged': False, 'interface': u'bond0', 'ipv6gateway': '::', 'gateway': '172.16.0.1'}, 'Blue': {'iface': 'Blue', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.6']}, 'Infrastructure': {'iface': 'Infrastructure', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.4']}, 'migration': {'iface': u'bond0.5', 'addr': '', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'bridged': False, 'interface': u'bond0.5', 'ipv6gateway': '::', 'gateway': ''}, 'Red': {'iface': 'Red', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.3']}}, 'bridges': {'Blue': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.6']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'Infrastructure': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.4']}, 'Red': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.3']}}, 'uuid': '4C4C4544-0059-5310-8031-B9C04F445A31', 'lastClientIface': 'bond0', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:5F', 'MASTER': 'bond0', 'DEVICE': 'em1', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:5F', 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:60', 'MASTER': 'bond0', 'DEVICE': 'em2', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:60', 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '12', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,lahf_lm,ida,arat,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:93188a588127', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '255.255.0.0', 'addr': '172.16.10.1', 'slaves': ['em1', 'em2'], 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.10.1', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond0', 'MTU': '1500', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32057', 'cpuSpeed': '1900.070', 'version_name': 'Snow Man', 'vlans': {'bond0.3': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Red', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 3, 'mtu': '1500'}, 'bond0.4': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Infrastructure', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 4, 'mtu': '1500'}, 'bond0.5': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'DEVICE': 'bond0.5', 'NM_CONTROLLED': 'no', 'VLAN': 'yes', 'DEFROUTE': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 5, 'mtu': '1500'}, 'bond0.6': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Blue', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.6', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 6, 'mtu': '1500'}}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '24', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.10.1'}} >Thread-14::DEBUG::2014-04-11 22:53:39,381::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getHardwareInfo with () {} >Thread-14::DEBUG::2014-04-11 22:53:39,383::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R420', 'systemSerialNumber': '9YS1DZ1', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0059-5310-8031-B9C04F445A31', 'systemManufacturer': 'Dell Inc.'}} >Thread-14::DEBUG::2014-04-11 22:53:39,462::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getCapabilities with () {} >Thread-14::DEBUG::2014-04-11 22:53:39,470::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,482::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,521::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,530::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,537::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,546::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,553::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.5' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,562::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,566::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.6' (cwd None) >Thread-14::DEBUG::2014-04-11 22:53:39,576::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:53:39,598::caps::458::root::(_getKeyPackages) rpm package glusterfs-rdma not found >Thread-14::DEBUG::2014-04-11 22:53:39,599::caps::458::root::(_getKeyPackages) rpm package glusterfs-fuse not found >Thread-14::DEBUG::2014-04-11 22:53:39,601::caps::458::root::(_getKeyPackages) rpm package gluster-swift not found >Thread-14::DEBUG::2014-04-11 22:53:39,608::caps::458::root::(_getKeyPackages) rpm package gluster-swift-object not found >Thread-14::DEBUG::2014-04-11 22:53:39,609::caps::458::root::(_getKeyPackages) rpm package glusterfs not found >Thread-14::DEBUG::2014-04-11 22:53:39,609::caps::458::root::(_getKeyPackages) rpm package gluster-swift-plugin not found >Thread-14::DEBUG::2014-04-11 22:53:39,619::caps::458::root::(_getKeyPackages) rpm package gluster-swift-account not found >Thread-14::DEBUG::2014-04-11 22:53:39,620::caps::458::root::(_getKeyPackages) rpm package gluster-swift-proxy not found >Thread-14::DEBUG::2014-04-11 22:53:39,620::caps::458::root::(_getKeyPackages) rpm package gluster-swift-doc not found >Thread-14::DEBUG::2014-04-11 22:53:39,621::caps::458::root::(_getKeyPackages) rpm package glusterfs-server not found >Thread-14::DEBUG::2014-04-11 22:53:39,621::caps::458::root::(_getKeyPackages) rpm package gluster-swift-container not found >Thread-14::DEBUG::2014-04-11 22:53:39,622::caps::458::root::(_getKeyPackages) rpm package glusterfs-geo-replication not found >Thread-14::DEBUG::2014-04-11 22:53:39,630::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:93188a588127'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395791995.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '12.git49cacf0.el6', 'buildtime': 1397150575L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.7', 'buildtime': 1396537962L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz', 'hooks': {'after_vm_start': {'50_promisc': {'md5': '5ea469fa06dd7cd1e52508672a7519db'}}, 'before_vm_migrate_source': {'50_sriov': {'md5': '47a59103b7c8b1439ae17dcd5ed197cd'}, '50_hostusb': {'md5': 'e50a0975acd8bafd0052b65a787c2fdd'}, '50_scratchpad': {'md5': 'be90995f959c60c5110c6a0bef67c7b8'}, '50_floppy': {'md5': 'c68fb91d44105c8b6f2d80566314027a'}}, 'after_device_destroy': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_vm_destroy': {'50_promisc': {'md5': 'dae0434ea8ca2492422a6fde5cc47a1e'}}, 'before_vm_start': {'50_sriov': {'md5': '3ebc60cd2e4eb089820102285fad7c45'}, '50_vmfex': {'md5': '9f5abb892ddb6b3daa779985d38d9f55'}, '50_qos': {'md5': 'cdf6f89d277a886691812ce1ec2fe105'}, '50_scratchpad': {'md5': '178171eb62ba95fc548e2450013f5ad5'}, '50_numa': {'md5': '5008c2826714ac5b63748780aabd2f25'}, '50_floppy': {'md5': '202fe18705a7d4c50c40c126e8f8dbe8'}, '50_qemucmdline': {'md5': 'a884929ad6f5eb039887157288867409'}, '50_pincpu': {'md5': '0b5fb99ff0e7acb9ad534b87c02c59e3'}, '50_hugepages': {'md5': '6713d703f9d316f1664145b700cd7904'}, '50_fileinject': {'md5': '290dafd45601e010e0f496fa5b889a76'}, '50_macspoof': {'md5': '25deea559772719b90117f2e1009d823'}, '60_checkimages': {'md5': 'abd32c723a90ab2816e2163f5e02faf7'}, '10_faqemu': {'md5': 'e3cb338be6b9075a04bed0e7400ab7e2'}, '50_directlun': {'md5': 'dba5d80fdb7119ee0f4a40c26eac8448'}, '50_smbios': {'md5': '1a6bd8eddc3a86943e558bc4d3e73bd6'}, '50_vmdisk': {'md5': '67617c6e2d796388ddc6c065a97f714e'}, '50_hostusb': {'md5': '21c58568f3eae4f0d091005e0eff891a'}, '50_isolatedprivatevlan': {'md5': '0a25d7cd0b807ea5e0dfc3318ee98c91'}}, 'after_device_migrate_destination': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_device_create': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'after_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': 'ec28d5c17500ef0cf8f1a6393ac58506'}}, 'before_vm_migrate_destination': {'50_vmfex': {'md5': 'eac2633f47fb309b4a5b7fe76531baec'}, '50_hugepages': {'md5': '6ddc484746297308b9f9acc5fcabaf10'}, '50_directlun': {'md5': '8a29ce41de079630cb58d48e280d55c2'}}, 'after_nic_hotunplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '356db41e728db63bf6f7b28c5cfcbf70'}}, 'before_nic_hotplug': {'openstacknet_utils.py': {'md5': 'feebf44191d76fbc26e579a6323ce55c'}, '50_openstacknet': {'md5': '1d72e12a9eea561591516bec754cc725'}, '50_macspoof': {'md5': 'ca9c8ade572d4b5b4390755d4196c416'}}, 'after_vm_destroy': {'50_sriov': {'md5': '5890b2899d450495a62f3611fad3c526'}, '50_directlun': {'md5': '6d05c19e68a141eeb7ee5981d68c0cfa'}, '50_hostusb': {'md5': '0a37bf0a26546e3ab5e424b13653ce2b'}, '50_hugepages': {'md5': '4a980b78f8ee908bf2c61ea9bf156f4a'}, '50_scratchpad': {'md5': '316d21a669f97e83b485b45ab0b23eee'}}}, 'cpuSockets': '2', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0', 'addr': '172.16.10.1', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'bridged': False, 'interface': u'bond0', 'ipv6gateway': '::', 'gateway': '172.16.0.1'}, 'Blue': {'iface': 'Blue', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.6']}, 'Infrastructure': {'iface': 'Infrastructure', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.4']}, 'migration': {'iface': u'bond0.5', 'addr': '', 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'bridged': False, 'interface': u'bond0.5', 'ipv6gateway': '::', 'gateway': ''}, 'Red': {'iface': 'Red', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'gateway': '', 'netmask': '', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['bond0.3']}}, 'bridges': {'Blue': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Blue', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.6']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}, 'Infrastructure': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Infrastructure', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.4']}, 'Red': {'addr': '', 'cfg': {'DEFROUTE': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'Red', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': ['bond0.3']}}, 'uuid': '4C4C4544-0059-5310-8031-B9C04F445A31', 'lastClientIface': 'bond0', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:5F', 'MASTER': 'bond0', 'DEVICE': 'em1', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:5F', 'speed': 1000, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': 'F0:1F:AF:DE:00:60', 'MASTER': 'bond0', 'DEVICE': 'em2', 'STP': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'permhwaddr': 'F0:1F:AF:DE:00:60', 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '12', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,lahf_lm,ida,arat,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:93188a588127', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '255.255.0.0', 'addr': '172.16.10.1', 'slaves': ['em1', 'em2'], 'hwaddr': 'f0:1f:af:de:00:5f', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.10.1', 'GATEWAY': '172.16.0.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond0', 'MTU': '1500', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '32057', 'cpuSpeed': '1900.070', 'version_name': 'Snow Man', 'vlans': {'bond0.3': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Red', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 3, 'mtu': '1500'}, 'bond0.4': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Infrastructure', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 4, 'mtu': '1500'}, 'bond0.5': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'DEVICE': 'bond0.5', 'NM_CONTROLLED': 'no', 'VLAN': 'yes', 'DEFROUTE': 'no', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 5, 'mtu': '1500'}, 'bond0.6': {'netmask': '', 'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'Blue', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'bond0.6', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::f21f:afff:fede:5f/64'], 'vlanid': 6, 'mtu': '1500'}}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '24', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '172.16.10.1'}} >Thread-14::DEBUG::2014-04-11 22:53:39,697::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call getHardwareInfo with () {} >Thread-14::DEBUG::2014-04-11 22:53:39,699::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R420', 'systemSerialNumber': '9YS1DZ1', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0059-5310-8031-B9C04F445A31', 'systemManufacturer': 'Dell Inc.'}} >Thread-14::DEBUG::2014-04-11 22:53:39,894::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [7108a818] >Thread-14::DEBUG::2014-04-11 22:53:39,895::task::595::TaskManager.Task::(_updateState) Task=`c0c94951-72b1-433c-9dfe-df12b3e00e2d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:53:39,895::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', conList=[{'connection': 'ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c', 'port': ''}, {'connection': 'ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/exports', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54', 'port': ''}, {'connection': 'ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'deb06d0b-14c6-490e-8fbb-552861511109', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/infrastructure', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac', 'port': ''}], options=None) >Thread-14::WARNING::2014-04-11 22:53:39,906::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage already exists >Thread-14::DEBUG::2014-04-11 22:53:39,907::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh02.suprtekstic.com:/var/lib/exports/storage /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage' (cwd None) >Thread-15::DEBUG::2014-04-11 22:54:06,246::API::1134::vds::(fenceNode) fenceNode(addr=172.16.10.4,port=,agent=ipmilan,user=root,passwd=XXXX,action=status,secure=,options=lanplus) >Thread-15::DEBUG::2014-04-11 22:54:06,337::API::1160::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=172.16.10.4 >login=root >action=status >passwd=XXXX >lanplus out Getting status of IPMI:172.16.10.4...Chassis power = On >Done > err >Thread-15::DEBUG::2014-04-11 22:54:06,389::API::1134::vds::(fenceNode) fenceNode(addr=172.16.10.4,port=,agent=ipmilan,user=root,passwd=XXXX,action=status,secure=,options=lanplus) >Thread-15::DEBUG::2014-04-11 22:54:06,445::API::1160::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=172.16.10.4 >login=root >action=status >passwd=XXXX >lanplus out Getting status of IPMI:172.16.10.4...Chassis power = On >Done > err >Thread-14::DEBUG::2014-04-11 22:54:44,016::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage >Thread-14::DEBUG::2014-04-11 22:54:44,037::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('a033c2ac-0d01-490c-9552-99ca53d6a64a',) >Thread-14::DEBUG::2014-04-11 22:54:44,038::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain} >Thread-14::WARNING::2014-04-11 22:54:44,044::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage already exists >Thread-14::DEBUG::2014-04-11 22:54:44,045::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh03.suprtekstic.com:/var/lib/exports/storage /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage' (cwd None) >Thread-14::DEBUG::2014-04-11 22:54:44,271::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage >Thread-14::DEBUG::2014-04-11 22:54:44,286::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('423b56b3-3148-468c-8c28-4c96fddbc1c8',) >Thread-14::DEBUG::2014-04-11 22:54:44,288::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain} >Thread-14::DEBUG::2014-04-11 22:54:44,331::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/exports /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_exports' (cwd None) >Thread-14::ERROR::2014-04-11 22:55:36,659::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect > self._mount.mount(self.options, self._vfsType) > File "/usr/share/vdsm/storage/mount.py", line 222, in mount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-14::ERROR::2014-04-11 22:55:36,705::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer > conObj.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect > return self._mountCon.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect > raise e >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-14::DEBUG::2014-04-11 22:55:36,705::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain} >Thread-14::WARNING::2014-04-11 22:55:36,713::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage already exists >Thread-14::DEBUG::2014-04-11 22:55:36,714::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh04.suprtekstic.com:/var/lib/exports/storage /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage' (cwd None) >Thread-14::DEBUG::2014-04-11 22:55:36,957::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage >Thread-14::DEBUG::2014-04-11 22:55:36,969::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('b7663d70-e658-41fa-b9f0-8da83c9eddce',) >Thread-14::DEBUG::2014-04-11 22:55:36,969::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain} >Thread-14::DEBUG::2014-04-11 22:55:36,977::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtistg01.suprtekstic.com:/storage/infrastructure /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure' (cwd None) >Thread-14::ERROR::2014-04-11 22:56:29,307::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect > self._mount.mount(self.options, self._vfsType) > File "/usr/share/vdsm/storage/mount.py", line 222, in mount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-14::ERROR::2014-04-11 22:56:29,309::hsm::2379::Storage.HSM::(connectStorageServer) Could not connect to storageServer >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2376, in connectStorageServer > conObj.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect > return self._mountCon.connect() > File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect > raise e >MountError: (32, ';mount.nfs: Failed to resolve server ashtistg01.suprtekstic.com: Name or service not known\n') >Thread-14::DEBUG::2014-04-11 22:56:29,309::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain} >Thread-14::WARNING::2014-04-11 22:56:29,317::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso already exists >Thread-14::DEBUG::2014-04-11 22:56:29,317::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh01.suprtekstic.com:/var/lib/exports/iso /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso' (cwd None) >Thread-14::DEBUG::2014-04-11 22:56:29,391::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso >Thread-14::DEBUG::2014-04-11 22:56:29,394::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e2ffd939-380e-4fbb-8a23-5a0e69c84a87',) >Thread-14::DEBUG::2014-04-11 22:56:29,395::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain} >Thread-14::WARNING::2014-04-11 22:56:29,403::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage already exists >Thread-14::DEBUG::2014-04-11 22:56:29,404::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 ashtivh01.suprtekstic.com:/var/lib/exports/storage /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage' (cwd None) >Thread-14::DEBUG::2014-04-11 22:56:29,461::hsm::2318::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage >Thread-14::DEBUG::2014-04-11 22:56:29,465::hsm::2342::Storage.HSM::(__prefetchDomains) Found SD uuids: ('21619c8e-99ea-4813-be02-d708971e5393',) >Thread-14::DEBUG::2014-04-11 22:56:29,465::hsm::2398::Storage.HSM::(connectStorageServer) knownSDs: {423b56b3-3148-468c-8c28-4c96fddbc1c8: storage.nfsSD.findDomain, b7663d70-e658-41fa-b9f0-8da83c9eddce: storage.nfsSD.findDomain, e2ffd939-380e-4fbb-8a23-5a0e69c84a87: storage.nfsSD.findDomain, a033c2ac-0d01-490c-9552-99ca53d6a64a: storage.nfsSD.findDomain, 21619c8e-99ea-4813-be02-d708971e5393: storage.nfsSD.findDomain} >Thread-14::INFO::2014-04-11 22:56:29,466::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]} >Thread-14::DEBUG::2014-04-11 22:56:29,466::task::1185::TaskManager.Task::(prepare) Task=`c0c94951-72b1-433c-9dfe-df12b3e00e2d`::finished: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]} >Thread-14::DEBUG::2014-04-11 22:56:29,467::task::595::TaskManager.Task::(_updateState) Task=`c0c94951-72b1-433c-9dfe-df12b3e00e2d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:56:29,467::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:56:29,468::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:56:29,468::task::990::TaskManager.Task::(_decref) Task=`c0c94951-72b1-433c-9dfe-df12b3e00e2d`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:56:29,627::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:56:29,628::task::595::TaskManager.Task::(_updateState) Task=`106510a9-25a0-4951-83f3-95def0ec5828`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:56:29,628::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, msdUUID='a033c2ac-0d01-490c-9552-99ca53d6a64a', masterVersion=38, domainsMap=None, options=None) >Thread-14::DEBUG::2014-04-11 22:56:29,629::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`02bb6b22-94ec-4909-8039-766ead921575`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1008' at 'connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:56:29,630::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:56:29,630::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:56:29,631::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`02bb6b22-94ec-4909-8039-766ead921575`::Granted request >Thread-14::DEBUG::2014-04-11 22:56:29,632::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`526636e4-6be0-490c-8ae6-3155207ed1fd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1056' at '_connectStoragePool' >Thread-14::DEBUG::2014-04-11 22:56:29,632::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:56:29,633::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:56:29,633::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`526636e4-6be0-490c-8ae6-3155207ed1fd`::Granted request >Thread-14::INFO::2014-04-11 22:56:29,634::sp::133::Storage.StoragePool::(setBackend) updating pool a106ab81-9d5f-49c1-aeaf-832a137b708c backend from type NoneType instance 0x3404972210 to type StoragePoolDiskBackend instance 0x7fc3ec0b4ea8 >Thread-14::INFO::2014-04-11 22:56:29,634::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c with master domain: a033c2ac-0d01-490c-9552-99ca53d6a64a (ver = 38) >Thread-14::DEBUG::2014-04-11 22:56:29,635::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,635::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,636::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,636::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,636::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,637::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,637::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 22:56:29,638::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:56:29,638::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 22:56:29,638::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:56:29,639::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 22:56:29,639::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 22:56:29,667::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:56:29,668::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 22:56:29,822::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:56:29,823::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,824::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,824::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,824::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,825::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,825::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 22:56:29,826::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 22:56:30,149::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::DEBUG::2014-04-11 22:56:30,434::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-14::DEBUG::2014-04-11 22:56:30,459::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >Thread-14::DEBUG::2014-04-11 22:56:30,476::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-14::DEBUG::2014-04-11 22:56:30,477::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS' >Thread-14::DEBUG::2014-04-11 22:56:30,488::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS' >Thread-14::DEBUG::2014-04-11 22:56:30,489::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain a033c2ac-0d01-490c-9552-99ca53d6a64a verified, version 38 >Thread-14::DEBUG::2014-04-11 22:56:30,489::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-14::DEBUG::2014-04-11 22:56:30,490::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 22:56:30,505::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >Thread-14::INFO::2014-04-11 22:56:30,506::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-22::DEBUG::2014-04-11 22:56:30,507::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-14::DEBUG::2014-04-11 22:56:30,507::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `423b56b3-3148-468c-8c28-4c96fddbc1c8` >Thread-14::INFO::2014-04-11 22:56:30,509::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-24::DEBUG::2014-04-11 22:56:30,511::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-14::DEBUG::2014-04-11 22:56:30,511::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` >Thread-14::INFO::2014-04-11 22:56:30,514::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-14::DEBUG::2014-04-11 22:56:30,517::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `b7663d70-e658-41fa-b9f0-8da83c9eddce` >Thread-14::INFO::2014-04-11 22:56:30,518::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-29::DEBUG::2014-04-11 22:56:30,520::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-14::DEBUG::2014-04-11 22:56:30,528::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `3406665e-4adc-4fd4-aa1e-037547b29adb` >Thread-33::DEBUG::2014-04-11 22:56:30,529::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-14::INFO::2014-04-11 22:56:30,529::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-33::ERROR::2014-04-11 22:56:30,531::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:56:30,536::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:56:30,539::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2014-04-11 22:56:30,541::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-39::DEBUG::2014-04-11 22:56:30,541::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-14::DEBUG::2014-04-11 22:56:30,543::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `f3b51811-4a7f-43af-8633-322b3db23c48` >Thread-14::INFO::2014-04-11 22:56:30,549::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-39::ERROR::2014-04-11 22:56:30,557::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-43::DEBUG::2014-04-11 22:56:30,563::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-39::ERROR::2014-04-11 22:56:30,563::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-14::DEBUG::2014-04-11 22:56:30,565::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `a033c2ac-0d01-490c-9552-99ca53d6a64a` >Thread-24::DEBUG::2014-04-11 22:56:30,568::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-39::DEBUG::2014-04-11 22:56:30,572::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-14::INFO::2014-04-11 22:56:30,574::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 21619c8e-99ea-4813-be02-d708971e5393 >Thread-14::DEBUG::2014-04-11 22:56:30,593::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `a106ab81-9d5f-49c1-aeaf-832a137b708c` started monitoring domain `21619c8e-99ea-4813-be02-d708971e5393` >Thread-46::DEBUG::2014-04-11 22:56:30,594::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 21619c8e-99ea-4813-be02-d708971e5393 >Thread-14::DEBUG::2014-04-11 22:56:30,599::misc::772::SamplingMethod::(__call__) Returning last result >Thread-33::DEBUG::2014-04-11 22:56:30,602::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-29::DEBUG::2014-04-11 22:56:30,605::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-39::DEBUG::2014-04-11 22:56:30,606::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-14::DEBUG::2014-04-11 22:56:30,650::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >Thread-14::DEBUG::2014-04-11 22:56:30,657::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-14::DEBUG::2014-04-11 22:56:30,660::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 423b56b3-3148-468c-8c28-4c96fddbc1c8 e2ffd939-380e-4fbb-8a23-5a0e69c84a87 b7663d70-e658-41fa-b9f0-8da83c9eddce 3406665e-4adc-4fd4-aa1e-037547b29adb f3b51811-4a7f-43af-8633-322b3db23c48 21619c8e-99ea-4813-be02-d708971e5393' (cwd None) >Thread-46::DEBUG::2014-04-11 22:56:30,696::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393 >Thread-43::DEBUG::2014-04-11 22:56:30,696::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 22:56:30,735::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000491518 s, 1.5 MB/s\n'; <rc> = 0 >Thread-43::DEBUG::2014-04-11 22:56:30,745::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-43::DEBUG::2014-04-11 22:56:30,757::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain a033c2ac-0d01-490c-9552-99ca53d6a64a changed its status to Valid >Thread-43::INFO::2014-04-11 22:56:30,758::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-53::DEBUG::2014-04-11 22:56:30,758::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-43::DEBUG::2014-04-11 22:56:30,761::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-53::DEBUG::2014-04-11 22:56:30,762::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-53::DEBUG::2014-04-11 22:56:30,789::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-54::INFO::2014-04-11 22:56:30,790::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-22::DEBUG::2014-04-11 22:56:30,843::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-24::DEBUG::2014-04-11 22:56:30,868::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-22::DEBUG::2014-04-11 22:56:30,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955'] >Thread-24::DEBUG::2014-04-11 22:56:30,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_Storage', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=e2ffd939-380e-4fbb-8a23-5a0e69c84a87', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=aa653ed432ff22d61d7b0ce5019a22c0a8bb40dd'] >Thread-22::DEBUG::2014-04-11 22:56:30,889::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-24::DEBUG::2014-04-11 22:56:30,890::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-22::DEBUG::2014-04-11 22:56:30,893::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '423b56b3-3148-468c-8c28-4c96fddbc1c8_imageNS' >Thread-24::DEBUG::2014-04-11 22:56:30,895::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87_imageNS' >Thread-22::DEBUG::2014-04-11 22:56:30,897::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '423b56b3-3148-468c-8c28-4c96fddbc1c8_volumeNS' >Thread-24::DEBUG::2014-04-11 22:56:30,898::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87_volumeNS' >Thread-29::DEBUG::2014-04-11 22:56:30,917::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-33::DEBUG::2014-04-11 22:56:30,979::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-39::DEBUG::2014-04-11 22:56:30,983::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-29::DEBUG::2014-04-11 22:56:30,991::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40'] >Thread-39::WARNING::2014-04-11 22:56:31,015::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-33::WARNING::2014-04-11 22:56:31,015::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-46::DEBUG::2014-04-11 22:56:31,016::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-14::DEBUG::2014-04-11 22:56:31,030::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found\n Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8\n Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found\n Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87\n Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found\n Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce\n Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found\n Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393\n'; <rc> = 5 >Thread-29::DEBUG::2014-04-11 22:56:31,064::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-14::WARNING::2014-04-11 22:56:31,080::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "423b56b3-3148-468c-8c28-4c96fddbc1c8" not found', ' Skipping volume group 423b56b3-3148-468c-8c28-4c96fddbc1c8', ' Volume group "e2ffd939-380e-4fbb-8a23-5a0e69c84a87" not found', ' Skipping volume group e2ffd939-380e-4fbb-8a23-5a0e69c84a87', ' Volume group "b7663d70-e658-41fa-b9f0-8da83c9eddce" not found', ' Skipping volume group b7663d70-e658-41fa-b9f0-8da83c9eddce', ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb', ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48', ' Volume group "21619c8e-99ea-4813-be02-d708971e5393" not found', ' Skipping volume group 21619c8e-99ea-4813-be02-d708971e5393'] >Thread-29::DEBUG::2014-04-11 22:56:31,083::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'b7663d70-e658-41fa-b9f0-8da83c9eddce_imageNS' >Thread-14::DEBUG::2014-04-11 22:56:31,109::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29::DEBUG::2014-04-11 22:56:31,118::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'b7663d70-e658-41fa-b9f0-8da83c9eddce_volumeNS' >Thread-46::DEBUG::2014-04-11 22:56:31,135::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2'] >Thread-39::ERROR::2014-04-11 22:56:31,355::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-46::DEBUG::2014-04-11 22:56:31,395::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-39::ERROR::2014-04-11 22:56:31,414::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-46::DEBUG::2014-04-11 22:56:31,431::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '21619c8e-99ea-4813-be02-d708971e5393_imageNS' >Thread-14::INFO::2014-04-11 22:56:31,434::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-39::DEBUG::2014-04-11 22:56:31,445::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain f3b51811-4a7f-43af-8633-322b3db23c48 changed its status to Invalid >Thread-14::INFO::2014-04-11 22:56:31,457::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-46::DEBUG::2014-04-11 22:56:31,467::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '21619c8e-99ea-4813-be02-d708971e5393_volumeNS' >Thread-73::DEBUG::2014-04-11 22:56:31,467::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-14::INFO::2014-04-11 22:56:31,475::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-73::DEBUG::2014-04-11 22:56:31,490::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-14::INFO::2014-04-11 22:56:31,493::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393 to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/21619c8e-99ea-4813-be02-d708971e5393 >Thread-73::DEBUG::2014-04-11 22:56:31,514::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-14::INFO::2014-04-11 22:56:31,514::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-14::INFO::2014-04-11 22:56:31,532::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a to /rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/mastersd >Thread-14::DEBUG::2014-04-11 22:56:31,553::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 22:56:31,570::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 22:56:31,580::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 22:56:31,590::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 22:56:31,612::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-33::ERROR::2014-04-11 22:56:31,625::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:56:31,630::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-33::ERROR::2014-04-11 22:56:31,639::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:56:31,643::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-33::DEBUG::2014-04-11 22:56:31,657::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 3406665e-4adc-4fd4-aa1e-037547b29adb changed its status to Invalid >Thread-14::DEBUG::2014-04-11 22:56:31,662::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-75::DEBUG::2014-04-11 22:56:31,673::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-14::INFO::2014-04-11 22:56:31,681::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-75::DEBUG::2014-04-11 22:56:31,685::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-14::DEBUG::2014-04-11 22:56:31,692::task::1185::TaskManager.Task::(prepare) Task=`106510a9-25a0-4951-83f3-95def0ec5828`::finished: True >Thread-75::DEBUG::2014-04-11 22:56:31,701::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-14::DEBUG::2014-04-11 22:56:31,704::task::595::TaskManager.Task::(_updateState) Task=`106510a9-25a0-4951-83f3-95def0ec5828`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:56:31,715::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:56:31,725::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:56:31,737::task::990::TaskManager.Task::(_decref) Task=`106510a9-25a0-4951-83f3-95def0ec5828`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:56:31,826::task::595::TaskManager.Task::(_updateState) Task=`872ca3a1-de16-48aa-ba43-65a07a77546d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:56:31,838::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:56:31,869::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000491518', 'lastCheck': '1.1', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:56:31,878::task::1185::TaskManager.Task::(prepare) Task=`872ca3a1-de16-48aa-ba43-65a07a77546d`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000491518', 'lastCheck': '1.1', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.3', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:56:31,889::task::595::TaskManager.Task::(_updateState) Task=`872ca3a1-de16-48aa-ba43-65a07a77546d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:56:31,899::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:56:31,909::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:56:31,919::task::990::TaskManager.Task::(_decref) Task=`872ca3a1-de16-48aa-ba43-65a07a77546d`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:56:31,929::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:56:32,003::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:56:32,038::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000248225 s, 1.5 MB/s\n'; <rc> = 0 >Thread-24::DEBUG::2014-04-11 22:56:32,073::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 changed its status to Valid >Thread-77::DEBUG::2014-04-11 22:56:32,083::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-77::DEBUG::2014-04-11 22:56:32,091::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-77::DEBUG::2014-04-11 22:56:32,103::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-78::INFO::2014-04-11 22:56:32,105::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-22::DEBUG::2014-04-11 22:56:32,112::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000456571 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:56:32,136::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-22::DEBUG::2014-04-11 22:56:32,145::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 changed its status to Valid >Thread-79::DEBUG::2014-04-11 22:56:32,153::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-22::INFO::2014-04-11 22:56:32,154::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-79::DEBUG::2014-04-11 22:56:32,161::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-22::DEBUG::2014-04-11 22:56:32,164::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-79::DEBUG::2014-04-11 22:56:32,173::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-80::INFO::2014-04-11 22:56:32,175::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-29::DEBUG::2014-04-11 22:56:32,196::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:56:32,226::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:56:32,240::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000367226 s, 2.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:56:32,250::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-29::DEBUG::2014-04-11 22:56:32,254::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain b7663d70-e658-41fa-b9f0-8da83c9eddce changed its status to Valid >Thread-46::DEBUG::2014-04-11 22:56:32,254::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000315664 s, 2.4 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-04-11 22:56:32,258::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-29::INFO::2014-04-11 22:56:32,259::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1) >Thread-81::DEBUG::2014-04-11 22:56:32,263::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-29::DEBUG::2014-04-11 22:56:32,265::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-81::DEBUG::2014-04-11 22:56:32,272::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-82::INFO::2014-04-11 22:56:32,274::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-46::DEBUG::2014-04-11 22:56:32,282::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-46::DEBUG::2014-04-11 22:56:32,282::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 21619c8e-99ea-4813-be02-d708971e5393 changed its status to Valid >Thread-83::DEBUG::2014-04-11 22:56:32,283::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-46::INFO::2014-04-11 22:56:32,284::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1) >Thread-83::DEBUG::2014-04-11 22:56:32,286::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-46::DEBUG::2014-04-11 22:56:32,288::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-83::DEBUG::2014-04-11 22:56:32,292::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >Thread-84::INFO::2014-04-11 22:56:32,293::clientIF::126::vds::(contEIOVms) vmContainerLock acquired >Thread-14::DEBUG::2014-04-11 22:56:33,094::BindingXMLRPC::1067::vds::(wrapper) client [172.16.10.1]::call setMOMPolicyParameters with ({'balloonEnabled': True, 'ksmEnabled': True},) {} flowID [7108a818] >Thread-14::DEBUG::2014-04-11 22:56:33,237::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}} >Thread-43::DEBUG::2014-04-11 22:56:40,819::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 22:56:40,836::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000699507 s, 1.1 MB/s\n'; <rc> = 0 >Thread-43::DEBUG::2014-04-11 22:56:40,850::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-43::INFO::2014-04-11 22:56:40,852::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-43::DEBUG::2014-04-11 22:56:40,852::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-39::ERROR::2014-04-11 22:56:41,468::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:56:41,469::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:56:41,469::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:56:41,470::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-14::DEBUG::2014-04-11 22:56:41,620::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:56:41,621::task::595::TaskManager.Task::(_updateState) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:56:41,622::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:56:41,623::task::866::TaskManager.Task::(_setError) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire > resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET) >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:56:41,650::task::885::TaskManager.Task::(_run) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::Task._run: 3345588b-6ade-4347-a769-d92a3d1c3fb8 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:56:41,650::task::1211::TaskManager.Task::(stop) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:56:41,650::task::990::TaskManager.Task::(_decref) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:56:41,651::task::1168::TaskManager.Task::(prepare) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100 >Thread-14::DEBUG::2014-04-11 22:56:41,651::task::1173::TaskManager.Task::(prepare) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:56:41,652::task::990::TaskManager.Task::(_decref) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:56:41,652::task::925::TaskManager.Task::(_doAbort) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:56:41,652::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:56:41,653::task::595::TaskManager.Task::(_updateState) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:56:41,653::task::550::TaskManager.Task::(__state_aborting) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:56:41,653::task::595::TaskManager.Task::(_updateState) Task=`3345588b-6ade-4347-a769-d92a3d1c3fb8`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:56:41,653::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:56:41,654::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:56:41,654::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-33::ERROR::2014-04-11 22:56:41,674::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:56:41,674::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:56:41,675::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:56:41,676::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:56:41,841::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:56:41,844::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:56:41,880::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:56:41,880::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:56:41,954::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:56:41,955::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:56:41,956::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:56:41,985::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:56:41,986::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-24::DEBUG::2014-04-11 22:56:42,109::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:56:42,122::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000360783 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:56:42,201::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:56:42,215::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000629432 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:56:42,218::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-22::INFO::2014-04-11 22:56:42,218::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-22::DEBUG::2014-04-11 22:56:42,219::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-29::DEBUG::2014-04-11 22:56:42,353::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:56:42,380::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000417906 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:56:42,385::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-29::INFO::2014-04-11 22:56:42,388::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1) >Thread-29::DEBUG::2014-04-11 22:56:42,389::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-46::DEBUG::2014-04-11 22:56:42,395::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:56:42,407::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000313101 s, 2.4 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:56:42,411::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-46::INFO::2014-04-11 22:56:42,412::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1) >Thread-46::DEBUG::2014-04-11 22:56:42,412::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-14::DEBUG::2014-04-11 22:56:48,555::task::595::TaskManager.Task::(_updateState) Task=`25a0efc7-e8c5-40db-8eda-6880ae3b6515`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:56:48,563::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:56:48,563::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000629432', 'lastCheck': '6.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000360783', 'lastCheck': '6.4', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000417906', 'lastCheck': '6.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.7', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000699507', 'lastCheck': '7.7', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000313101', 'lastCheck': '6.2', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:56:48,563::task::1185::TaskManager.Task::(prepare) Task=`25a0efc7-e8c5-40db-8eda-6880ae3b6515`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000629432', 'lastCheck': '6.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000360783', 'lastCheck': '6.4', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000417906', 'lastCheck': '6.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.7', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000699507', 'lastCheck': '7.7', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000313101', 'lastCheck': '6.2', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:56:48,563::task::595::TaskManager.Task::(_updateState) Task=`25a0efc7-e8c5-40db-8eda-6880ae3b6515`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:56:48,564::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:56:48,564::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:56:48,564::task::990::TaskManager.Task::(_decref) Task=`25a0efc7-e8c5-40db-8eda-6880ae3b6515`::ref 0 aborting False >Thread-43::DEBUG::2014-04-11 22:56:50,876::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 22:56:50,890::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000607005 s, 1.2 MB/s\n'; <rc> = 0 >Thread-43::DEBUG::2014-04-11 22:56:50,894::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-43::INFO::2014-04-11 22:56:50,895::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-43::DEBUG::2014-04-11 22:56:50,896::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-14::DEBUG::2014-04-11 22:56:51,787::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:56:51,788::task::595::TaskManager.Task::(_updateState) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:56:51,789::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:56:51,790::task::866::TaskManager.Task::(_setError) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire > resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET) >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:56:51,791::task::885::TaskManager.Task::(_run) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::Task._run: efe027f5-864e-4f41-b194-04d53c46e9e6 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:56:51,791::task::1211::TaskManager.Task::(stop) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:56:51,791::task::990::TaskManager.Task::(_decref) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:56:51,792::task::1168::TaskManager.Task::(prepare) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100 >Thread-14::DEBUG::2014-04-11 22:56:51,792::task::1173::TaskManager.Task::(prepare) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:56:51,793::task::990::TaskManager.Task::(_decref) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:56:51,793::task::925::TaskManager.Task::(_doAbort) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:56:51,793::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:56:51,794::task::595::TaskManager.Task::(_updateState) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:56:51,794::task::550::TaskManager.Task::(__state_aborting) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:56:51,794::task::595::TaskManager.Task::(_updateState) Task=`efe027f5-864e-4f41-b194-04d53c46e9e6`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:56:51,794::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:56:51,795::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:56:51,795::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-39::ERROR::2014-04-11 22:56:51,881::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:56:51,882::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:56:51,882::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:56:51,883::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:56:51,987::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:56:51,988::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:56:51,988::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:56:51,990::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-24::DEBUG::2014-04-11 22:56:52,164::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-39::DEBUG::2014-04-11 22:56:52,181::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:56:52,192::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-24::DEBUG::2014-04-11 22:56:52,203::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.00030025 s, 1.2 MB/s\n'; <rc> = 0 >Thread-39::ERROR::2014-04-11 22:56:52,255::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:56:52,260::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:56:52,276::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:56:52,287::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:56:52,291::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:56:52,341::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:56:52,344::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-22::DEBUG::2014-04-11 22:56:52,363::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:56:52,376::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000535976 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:56:52,379::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-22::INFO::2014-04-11 22:56:52,380::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-22::DEBUG::2014-04-11 22:56:52,380::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-29::DEBUG::2014-04-11 22:56:52,441::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:56:52,479::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000497721 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:56:52,491::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-29::INFO::2014-04-11 22:56:52,493::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1) >Thread-29::DEBUG::2014-04-11 22:56:52,495::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-46::DEBUG::2014-04-11 22:56:52,512::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:56:52,523::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000349473 s, 2.2 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:56:52,528::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-46::INFO::2014-04-11 22:56:52,528::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1) >Thread-46::DEBUG::2014-04-11 22:56:52,529::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-43::DEBUG::2014-04-11 22:57:00,917::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 22:57:00,931::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000515515 s, 1.5 MB/s\n'; <rc> = 0 >Thread-43::DEBUG::2014-04-11 22:57:00,936::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-43::INFO::2014-04-11 22:57:00,937::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-43::DEBUG::2014-04-11 22:57:00,937::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-14::DEBUG::2014-04-11 22:57:01,917::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:01,918::task::595::TaskManager.Task::(_updateState) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:01,919::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:57:01,920::task::866::TaskManager.Task::(_setError) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire > resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET) >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:57:01,921::task::885::TaskManager.Task::(_run) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::Task._run: fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:57:01,921::task::1211::TaskManager.Task::(stop) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:57:01,921::task::990::TaskManager.Task::(_decref) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:57:01,922::task::1168::TaskManager.Task::(prepare) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100 >Thread-14::DEBUG::2014-04-11 22:57:01,922::task::1173::TaskManager.Task::(prepare) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:57:01,923::task::990::TaskManager.Task::(_decref) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:57:01,923::task::925::TaskManager.Task::(_doAbort) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:57:01,923::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:01,923::task::595::TaskManager.Task::(_updateState) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:57:01,924::task::550::TaskManager.Task::(__state_aborting) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:57:01,924::task::595::TaskManager.Task::(_updateState) Task=`fc09cfa2-5e6d-4ebf-b12a-07f4a275c0fb`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:57:01,924::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:01,925::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:57:01,925::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-39::ERROR::2014-04-11 22:57:02,268::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:57:02,270::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:57:02,274::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:57:02,290::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-24::DEBUG::2014-04-11 22:57:02,309::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:57:02,321::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000338993 s, 1.1 MB/s\n'; <rc> = 0 >Thread-33::ERROR::2014-04-11 22:57:02,348::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:57:02,348::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:57:02,349::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:57:02,350::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-22::DEBUG::2014-04-11 22:57:02,406::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:57:02,420::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000613988 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:57:02,422::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-22::INFO::2014-04-11 22:57:02,423::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-22::DEBUG::2014-04-11 22:57:02,424::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-29::DEBUG::2014-04-11 22:57:02,525::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:57:02,560::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00051581 s, 1.5 MB/s\n'; <rc> = 0 >Thread-39::DEBUG::2014-04-11 22:57:02,579::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-29::DEBUG::2014-04-11 22:57:02,579::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-29::INFO::2014-04-11 22:57:02,592::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1) >Thread-39::WARNING::2014-04-11 22:57:02,598::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-29::DEBUG::2014-04-11 22:57:02,599::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-33::DEBUG::2014-04-11 22:57:02,604::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:57:02,633::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:57:02,656::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39::ERROR::2014-04-11 22:57:02,678::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:57:02,684::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::ERROR::2014-04-11 22:57:02,725::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:57:02,729::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-46::DEBUG::2014-04-11 22:57:02,748::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:57:02,761::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000388102 s, 1.9 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:57:02,783::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-46::INFO::2014-04-11 22:57:02,784::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1) >Thread-46::DEBUG::2014-04-11 22:57:02,784::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-14::DEBUG::2014-04-11 22:57:03,876::task::595::TaskManager.Task::(_updateState) Task=`c7b09696-4f2c-4c89-a29d-3c2fe150ecfc`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:03,877::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:57:03,878::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000613988', 'lastCheck': '1.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338993', 'lastCheck': '1.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00051581', 'lastCheck': '1.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000515515', 'lastCheck': '2.9', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000388102', 'lastCheck': '1.1', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:57:03,878::task::1185::TaskManager.Task::(prepare) Task=`c7b09696-4f2c-4c89-a29d-3c2fe150ecfc`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000613988', 'lastCheck': '1.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338993', 'lastCheck': '1.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00051581', 'lastCheck': '1.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000515515', 'lastCheck': '2.9', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000388102', 'lastCheck': '1.1', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:57:03,878::task::595::TaskManager.Task::(_updateState) Task=`c7b09696-4f2c-4c89-a29d-3c2fe150ecfc`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:03,879::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:03,879::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:03,879::task::990::TaskManager.Task::(_decref) Task=`c7b09696-4f2c-4c89-a29d-3c2fe150ecfc`::ref 0 aborting False >Thread-43::DEBUG::2014-04-11 22:57:10,958::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 22:57:10,971::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000523507 s, 1.4 MB/s\n'; <rc> = 0 >Thread-43::DEBUG::2014-04-11 22:57:10,980::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-43::INFO::2014-04-11 22:57:10,981::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-43::DEBUG::2014-04-11 22:57:10,982::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-14::DEBUG::2014-04-11 22:57:12,046::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:12,047::task::595::TaskManager.Task::(_updateState) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:12,048::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:57:12,049::task::866::TaskManager.Task::(_setError) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire > resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET) >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:57:12,050::task::885::TaskManager.Task::(_run) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::Task._run: 14d97fba-8715-4b83-ae07-0ff44c69e520 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:57:12,050::task::1211::TaskManager.Task::(stop) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:57:12,050::task::990::TaskManager.Task::(_decref) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:57:12,051::task::1168::TaskManager.Task::(prepare) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100 >Thread-14::DEBUG::2014-04-11 22:57:12,051::task::1173::TaskManager.Task::(prepare) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:57:12,051::task::990::TaskManager.Task::(_decref) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:57:12,052::task::925::TaskManager.Task::(_doAbort) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:57:12,052::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:12,052::task::595::TaskManager.Task::(_updateState) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:57:12,053::task::550::TaskManager.Task::(__state_aborting) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:57:12,053::task::595::TaskManager.Task::(_updateState) Task=`14d97fba-8715-4b83-ae07-0ff44c69e520`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:57:12,053::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:12,054::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:57:12,054::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-24::DEBUG::2014-04-11 22:57:12,349::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:57:12,361::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000321341 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:57:12,445::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:57:12,458::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000486723 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:57:12,461::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-22::INFO::2014-04-11 22:57:12,462::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-22::DEBUG::2014-04-11 22:57:12,462::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-29::DEBUG::2014-04-11 22:57:12,636::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:57:12,648::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000415012 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:57:12,651::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-29::INFO::2014-04-11 22:57:12,651::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1) >Thread-29::DEBUG::2014-04-11 22:57:12,652::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-39::ERROR::2014-04-11 22:57:12,698::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:57:12,698::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:57:12,699::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:57:12,700::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:57:12,736::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:57:12,736::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:57:12,737::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:57:12,738::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-46::DEBUG::2014-04-11 22:57:12,811::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:57:12,824::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000351933 s, 2.1 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:57:12,828::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-46::INFO::2014-04-11 22:57:12,829::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1) >Thread-46::DEBUG::2014-04-11 22:57:12,829::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-39::DEBUG::2014-04-11 22:57:12,966::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:57:12,968::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:57:13,003::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:57:13,003::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:57:13,010::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:57:13,012::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:57:13,012::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:57:13,042::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:57:13,042::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:57:19,200::task::595::TaskManager.Task::(_updateState) Task=`51f35ac7-1d14-473e-9106-232fe32ed366`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:19,215::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:57:19,216::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000486723', 'lastCheck': '6.8', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000321341', 'lastCheck': '6.9', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000415012', 'lastCheck': '6.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000523507', 'lastCheck': '8.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000351933', 'lastCheck': '6.4', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:57:19,216::task::1185::TaskManager.Task::(prepare) Task=`51f35ac7-1d14-473e-9106-232fe32ed366`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000486723', 'lastCheck': '6.8', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000321341', 'lastCheck': '6.9', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000415012', 'lastCheck': '6.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000523507', 'lastCheck': '8.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000351933', 'lastCheck': '6.4', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:57:19,217::task::595::TaskManager.Task::(_updateState) Task=`51f35ac7-1d14-473e-9106-232fe32ed366`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:19,217::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:19,217::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:19,218::task::990::TaskManager.Task::(_decref) Task=`51f35ac7-1d14-473e-9106-232fe32ed366`::ref 0 aborting False >Thread-43::DEBUG::2014-04-11 22:57:21,008::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 22:57:21,022::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000467351 s, 1.6 MB/s\n'; <rc> = 0 >Thread-43::DEBUG::2014-04-11 22:57:21,039::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-43::INFO::2014-04-11 22:57:21,040::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-43::DEBUG::2014-04-11 22:57:21,040::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('a033c2ac-0d01-490c-9552-99ca53d6a64a', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-14::DEBUG::2014-04-11 22:57:22,188::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:22,189::task::595::TaskManager.Task::(_updateState) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:22,190::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::ERROR::2014-04-11 22:57:22,191::task::866::TaskManager.Task::(_setError) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 621, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 615, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 493, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 264, in inquire > resource = sanlock.read_resource(self._leasesPath, SDM_LEASE_OFFSET) >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:57:22,193::task::885::TaskManager.Task::(_run) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::Task._run: ac3650a0-f5f8-4c65-b742-e802954f5d16 ('a106ab81-9d5f-49c1-aeaf-832a137b708c',) {} failed - stopping task >Thread-14::DEBUG::2014-04-11 22:57:22,194::task::1211::TaskManager.Task::(stop) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 22:57:22,194::task::990::TaskManager.Task::(_decref) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::ref 1 aborting True >Thread-14::INFO::2014-04-11 22:57:22,194::task::1168::TaskManager.Task::(prepare) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::aborting: Task is aborted: u"(111, 'Sanlock resource read failure', 'Connection refused')" - code 100 >Thread-14::DEBUG::2014-04-11 22:57:22,195::task::1173::TaskManager.Task::(prepare) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::Prepare: aborted: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-14::DEBUG::2014-04-11 22:57:22,195::task::990::TaskManager.Task::(_decref) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 22:57:22,195::task::925::TaskManager.Task::(_doAbort) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 22:57:22,196::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:22,196::task::595::TaskManager.Task::(_updateState) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 22:57:22,196::task::550::TaskManager.Task::(__state_aborting) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 22:57:22,197::task::595::TaskManager.Task::(_updateState) Task=`ac3650a0-f5f8-4c65-b742-e802954f5d16`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 22:57:22,197::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:22,197::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 22:57:22,198::dispatcher::68::Storage.Dispatcher.Protect::(run) (111, 'Sanlock resource read failure', 'Connection refused') >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SanlockException: (111, 'Sanlock resource read failure', 'Connection refused') >Thread-24::DEBUG::2014-04-11 22:57:22,391::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:57:22,403::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000368864 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:57:22,488::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:57:22,501::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000461848 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:57:22,504::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-22::INFO::2014-04-11 22:57:22,505::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-22::DEBUG::2014-04-11 22:57:22,505::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('423b56b3-3148-468c-8c28-4c96fddbc1c8', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-29::DEBUG::2014-04-11 22:57:22,677::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:57:22,689::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000441191 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:57:22,692::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-29::INFO::2014-04-11 22:57:22,692::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1) >Thread-29::DEBUG::2014-04-11 22:57:22,693::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('b7663d70-e658-41fa-b9f0-8da83c9eddce', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-46::DEBUG::2014-04-11 22:57:22,855::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:57:22,866::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000328371 s, 2.3 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:57:22,870::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False >Traceback (most recent call last): > File "/usr/share/vdsm/storage/clusterlock.py", line 223, in hasHostId > hostId, self._idsPath) >SanlockException: (111, 'Sanlock lockspace inquire failure', 'Connection refused') >Thread-46::INFO::2014-04-11 22:57:22,871::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1) >Thread-46::DEBUG::2014-04-11 22:57:22,872::domainMonitor::266::Storage.DomainMonitorThread::(_monitorDomain) Unable to issue the acquire host id 1 request for domain 21619c8e-99ea-4813-be02-d708971e5393 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 262, in _monitorDomain > self.domain.acquireHostId(self.hostId, async=True) > File "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/clusterlock.py", line 199, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('21619c8e-99ea-4813-be02-d708971e5393', SanlockException(111, 'Sanlock lockspace add failure', 'Connection refused')) >Thread-39::ERROR::2014-04-11 22:57:23,004::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:57:23,005::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:57:23,005::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:57:23,007::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:57:23,044::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:57:23,044::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:57:23,045::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:57:23,046::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:57:23,311::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:57:23,313::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-33::DEBUG::2014-04-11 22:57:23,353::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-39::ERROR::2014-04-11 22:57:23,355::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:57:23,360::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::WARNING::2014-04-11 22:57:23,364::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:57:23,365::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:57:23,395::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:57:23,395::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-43::DEBUG::2014-04-11 22:57:31,064::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 22:57:31,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000570109 s, 1.3 MB/s\n'; <rc> = 0 >Thread-43::INFO::2014-04-11 22:57:31,086::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-43::DEBUG::2014-04-11 22:57:31,086::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >Thread-14::DEBUG::2014-04-11 22:57:32,314::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:32,315::task::595::TaskManager.Task::(_updateState) Task=`5b62f416-800f-4558-8185-3a1e86db2c53`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:32,316::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 22:57:32,351::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 22:57:32,352::task::1185::TaskManager.Task::(prepare) Task=`5b62f416-800f-4558-8185-3a1e86db2c53`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 22:57:32,352::task::595::TaskManager.Task::(_updateState) Task=`5b62f416-800f-4558-8185-3a1e86db2c53`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:32,352::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:32,353::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:32,353::task::990::TaskManager.Task::(_decref) Task=`5b62f416-800f-4558-8185-3a1e86db2c53`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:32,418::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:32,423::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:32,427::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None) >Thread-14::DEBUG::2014-04-11 22:57:32,436::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`6baf763a-af73-41f5-865b-9c7a96010385`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart' >Thread-14::DEBUG::2014-04-11 22:57:32,439::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 22:57:32,444::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 22:57:32,447::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`6baf763a-af73-41f5-865b-9c7a96010385`::Granted request >Thread-14::DEBUG::2014-04-11 22:57:32,455::task::827::TaskManager.Task::(resourceAcquired) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive) >Thread-14::DEBUG::2014-04-11 22:57:32,458::task::990::TaskManager.Task::(_decref) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::ref 1 aborting False >Thread-14::INFO::2014-04-11 22:57:32,462::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-14::DEBUG::2014-04-11 22:57:32,465::task::1180::TaskManager.Task::(prepare) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Prepare: 1 jobs exist, move to acquiring >Thread-14::DEBUG::2014-04-11 22:57:32,469::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state preparing -> state acquiring >Thread-14::DEBUG::2014-04-11 22:57:32,470::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state acquiring -> state queued >Thread-14::DEBUG::2014-04-11 22:57:32,472::taskManager::50::TaskManager::(_queueTask) queuing task: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:32,476::taskManager::56::TaskManager::(_queueTask) task queued: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-5::DEBUG::2014-04-11 22:57:32,476::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-14::DEBUG::2014-04-11 22:57:32,481::task::1182::TaskManager.Task::(prepare) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::returning >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,482::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 9d3d90f5-8732-4a54-854e-573f3fdcd2de running: <bound method Task.commit of <storage.task.Task instance at 0x7fc3f818bc68>> with: None >Thread-14::DEBUG::2014-04-11 22:57:32,488::task::990::TaskManager.Task::(_decref) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::ref 0 aborting False >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,489::task::1194::TaskManager.Task::(commit) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::committing task: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,496::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state queued -> state running >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,503::task::905::TaskManager.Task::(_runJobs) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,510::task::331::TaskManager.Task::(run) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,517::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,523::misc::764::SamplingMethod::(__call__) Got in to sampling method >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,568::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 22:57:32,582::misc::772::SamplingMethod::(__call__) Returning last result >9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 22:57:32,604::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-24::DEBUG::2014-04-11 22:57:32,691::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:57:32,753::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000262429 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:57:32,824::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:57:32,860::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000421615 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::INFO::2014-04-11 22:57:32,875::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-22::DEBUG::2014-04-11 22:57:32,880::clusterlock::202::SANLock::(acquireHostId) Host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 successfully acquired (id: 1) >Thread-29::DEBUG::2014-04-11 22:57:32,950::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:57:32,983::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000441694 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::INFO::2014-04-11 22:57:32,996::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1) >Thread-29::DEBUG::2014-04-11 22:57:33,000::clusterlock::202::SANLock::(acquireHostId) Host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce successfully acquired (id: 1) >Thread-46::DEBUG::2014-04-11 22:57:33,010::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:57:33,022::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000240185 s, 3.1 MB/s\n'; <rc> = 0 >Thread-46::INFO::2014-04-11 22:57:33,028::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1) >Thread-46::DEBUG::2014-04-11 22:57:33,028::clusterlock::202::SANLock::(acquireHostId) Host id for domain 21619c8e-99ea-4813-be02-d708971e5393 successfully acquired (id: 1) >Thread-39::ERROR::2014-04-11 22:57:33,364::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:57:33,365::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:57:33,365::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:57:33,366::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:57:33,396::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:57:33,397::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:57:33,398::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:57:33,399::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 22:57:33,512::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:33,513::task::595::TaskManager.Task::(_updateState) Task=`f3a1968b-0e06-486c-af6a-32813115b9d1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:33,513::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:33,514::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:33,514::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:33,514::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:33,515::task::1185::TaskManager.Task::(prepare) Task=`f3a1968b-0e06-486c-af6a-32813115b9d1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:33,515::task::595::TaskManager.Task::(_updateState) Task=`f3a1968b-0e06-486c-af6a-32813115b9d1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:33,515::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:33,516::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:33,516::task::990::TaskManager.Task::(_decref) Task=`f3a1968b-0e06-486c-af6a-32813115b9d1`::ref 0 aborting False >Thread-33::DEBUG::2014-04-11 22:57:33,685::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-39::DEBUG::2014-04-11 22:57:33,694::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:57:33,696::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-33::WARNING::2014-04-11 22:57:33,696::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:57:33,702::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39::ERROR::2014-04-11 22:57:33,772::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:57:33,781::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::ERROR::2014-04-11 22:57:33,785::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:57:33,788::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:57:34,522::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:34,522::task::595::TaskManager.Task::(_updateState) Task=`699efc26-f273-473b-b950-ae00fe62d905`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:34,523::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:34,523::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:34,524::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:34,524::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:34,524::task::1185::TaskManager.Task::(prepare) Task=`699efc26-f273-473b-b950-ae00fe62d905`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:34,525::task::595::TaskManager.Task::(_updateState) Task=`699efc26-f273-473b-b950-ae00fe62d905`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:34,525::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:34,525::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:34,526::task::990::TaskManager.Task::(_decref) Task=`699efc26-f273-473b-b950-ae00fe62d905`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:34,537::task::595::TaskManager.Task::(_updateState) Task=`ff1d35e9-01d3-4234-9054-b2521de9ee0e`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:34,537::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:57:34,538::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000421615', 'lastCheck': '1.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000262429', 'lastCheck': '1.8', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000441694', 'lastCheck': '1.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '3.5', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000240185', 'lastCheck': '1.5', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:57:34,538::task::1185::TaskManager.Task::(prepare) Task=`ff1d35e9-01d3-4234-9054-b2521de9ee0e`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000421615', 'lastCheck': '1.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000262429', 'lastCheck': '1.8', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000441694', 'lastCheck': '1.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '3.5', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000240185', 'lastCheck': '1.5', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:57:34,539::task::595::TaskManager.Task::(_updateState) Task=`ff1d35e9-01d3-4234-9054-b2521de9ee0e`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:34,539::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:34,540::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:34,540::task::990::TaskManager.Task::(_decref) Task=`ff1d35e9-01d3-4234-9054-b2521de9ee0e`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:35,531::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:35,532::task::595::TaskManager.Task::(_updateState) Task=`b0e418ac-be48-428f-a286-c28ed9f912c3`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:35,532::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:35,533::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:35,533::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:35,533::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:35,534::task::1185::TaskManager.Task::(prepare) Task=`b0e418ac-be48-428f-a286-c28ed9f912c3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:35,534::task::595::TaskManager.Task::(_updateState) Task=`b0e418ac-be48-428f-a286-c28ed9f912c3`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:35,534::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:35,535::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:35,535::task::990::TaskManager.Task::(_decref) Task=`b0e418ac-be48-428f-a286-c28ed9f912c3`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:36,540::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:36,541::task::595::TaskManager.Task::(_updateState) Task=`f2a93513-efb6-4a68-9882-847b3db44d3f`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:36,541::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:36,541::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:36,542::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:36,542::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:36,542::task::1185::TaskManager.Task::(prepare) Task=`f2a93513-efb6-4a68-9882-847b3db44d3f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:36,543::task::595::TaskManager.Task::(_updateState) Task=`f2a93513-efb6-4a68-9882-847b3db44d3f`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:36,543::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:36,543::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:36,543::task::990::TaskManager.Task::(_decref) Task=`f2a93513-efb6-4a68-9882-847b3db44d3f`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:37,549::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:37,551::task::595::TaskManager.Task::(_updateState) Task=`08fab4f4-dd94-4891-87f9-e98849b13d4e`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:37,551::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:37,552::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:37,552::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:37,553::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:37,553::task::1185::TaskManager.Task::(prepare) Task=`08fab4f4-dd94-4891-87f9-e98849b13d4e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:37,553::task::595::TaskManager.Task::(_updateState) Task=`08fab4f4-dd94-4891-87f9-e98849b13d4e`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:37,554::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:37,554::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:37,554::task::990::TaskManager.Task::(_decref) Task=`08fab4f4-dd94-4891-87f9-e98849b13d4e`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:38,559::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:38,560::task::595::TaskManager.Task::(_updateState) Task=`9b8bf607-64e4-4d85-a77b-8caf26cebf26`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:38,561::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:38,561::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:38,561::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:38,562::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:38,562::task::1185::TaskManager.Task::(prepare) Task=`9b8bf607-64e4-4d85-a77b-8caf26cebf26`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:38,562::task::595::TaskManager.Task::(_updateState) Task=`9b8bf607-64e4-4d85-a77b-8caf26cebf26`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:38,563::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:38,563::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:38,563::task::990::TaskManager.Task::(_decref) Task=`9b8bf607-64e4-4d85-a77b-8caf26cebf26`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:39,568::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:39,568::task::595::TaskManager.Task::(_updateState) Task=`699d6680-b4a5-477d-a18a-146b95594b04`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:39,569::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:39,569::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:39,569::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:39,570::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:39,570::task::1185::TaskManager.Task::(prepare) Task=`699d6680-b4a5-477d-a18a-146b95594b04`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:39,570::task::595::TaskManager.Task::(_updateState) Task=`699d6680-b4a5-477d-a18a-146b95594b04`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:39,571::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:39,571::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:39,571::task::990::TaskManager.Task::(_decref) Task=`699d6680-b4a5-477d-a18a-146b95594b04`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:40,576::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:40,576::task::595::TaskManager.Task::(_updateState) Task=`fcb57fcb-64b5-48d5-9ca5-73cfef3e9fd0`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:40,577::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:40,577::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:40,577::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:40,577::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:40,578::task::1185::TaskManager.Task::(prepare) Task=`fcb57fcb-64b5-48d5-9ca5-73cfef3e9fd0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:40,578::task::595::TaskManager.Task::(_updateState) Task=`fcb57fcb-64b5-48d5-9ca5-73cfef3e9fd0`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:40,578::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:40,579::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:40,579::task::990::TaskManager.Task::(_decref) Task=`fcb57fcb-64b5-48d5-9ca5-73cfef3e9fd0`::ref 0 aborting False >Thread-43::DEBUG::2014-04-11 22:57:41,107::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 22:57:41,120::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000465291 s, 1.6 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:57:41,584::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:41,585::task::595::TaskManager.Task::(_updateState) Task=`c7687206-69c4-45f4-a87d-0651121cc0c3`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:41,586::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:41,586::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:41,586::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:41,587::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:41,587::task::1185::TaskManager.Task::(prepare) Task=`c7687206-69c4-45f4-a87d-0651121cc0c3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:41,587::task::595::TaskManager.Task::(_updateState) Task=`c7687206-69c4-45f4-a87d-0651121cc0c3`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:41,588::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:41,588::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:41,588::task::990::TaskManager.Task::(_decref) Task=`c7687206-69c4-45f4-a87d-0651121cc0c3`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:42,594::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:42,595::task::595::TaskManager.Task::(_updateState) Task=`9e1c4762-ace7-4c6b-b88b-5f2f8c29e419`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:42,596::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:42,596::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:42,596::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:42,597::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:42,597::task::1185::TaskManager.Task::(prepare) Task=`9e1c4762-ace7-4c6b-b88b-5f2f8c29e419`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:42,597::task::595::TaskManager.Task::(_updateState) Task=`9e1c4762-ace7-4c6b-b88b-5f2f8c29e419`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:42,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:42,598::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:42,598::task::990::TaskManager.Task::(_decref) Task=`9e1c4762-ace7-4c6b-b88b-5f2f8c29e419`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:57:42,800::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:57:42,813::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000355257 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:57:42,910::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:57:42,924::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000505038 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:57:43,027::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:57:43,059::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000477959 s, 1.6 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:57:43,091::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:57:43,102::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000370907 s, 2.0 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:57:43,604::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:43,618::task::595::TaskManager.Task::(_updateState) Task=`7cb64084-6b5f-4e85-9321-5aa034fd9b49`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:43,619::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:43,619::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:43,619::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:43,620::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:43,620::task::1185::TaskManager.Task::(prepare) Task=`7cb64084-6b5f-4e85-9321-5aa034fd9b49`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:43,620::task::595::TaskManager.Task::(_updateState) Task=`7cb64084-6b5f-4e85-9321-5aa034fd9b49`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:43,621::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:43,621::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:43,621::task::990::TaskManager.Task::(_decref) Task=`7cb64084-6b5f-4e85-9321-5aa034fd9b49`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:57:43,788::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:57:43,788::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-33::ERROR::2014-04-11 22:57:43,790::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-39::DEBUG::2014-04-11 22:57:43,791::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33::ERROR::2014-04-11 22:57:43,792::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:57:43,797::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-39::DEBUG::2014-04-11 22:57:43,814::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::DEBUG::2014-04-11 22:57:43,829::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:57:44,091::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:57:44,093::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-33::DEBUG::2014-04-11 22:57:44,129::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-39::ERROR::2014-04-11 22:57:44,132::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:57:44,136::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::WARNING::2014-04-11 22:57:44,141::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:57:44,141::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:57:44,171::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:57:44,172::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:57:44,628::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:44,629::task::595::TaskManager.Task::(_updateState) Task=`d8a17bf8-a475-4e44-b15c-f7d15390d6e1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:44,629::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:44,629::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:44,630::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:44,630::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:44,630::task::1185::TaskManager.Task::(prepare) Task=`d8a17bf8-a475-4e44-b15c-f7d15390d6e1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:44,631::task::595::TaskManager.Task::(_updateState) Task=`d8a17bf8-a475-4e44-b15c-f7d15390d6e1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:44,631::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:44,631::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:44,632::task::990::TaskManager.Task::(_decref) Task=`d8a17bf8-a475-4e44-b15c-f7d15390d6e1`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:45,637::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:45,637::task::595::TaskManager.Task::(_updateState) Task=`d5caadc1-96cc-49f7-b2af-84f4b63d7477`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:45,638::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:45,638::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:45,638::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:45,639::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:45,639::task::1185::TaskManager.Task::(prepare) Task=`d5caadc1-96cc-49f7-b2af-84f4b63d7477`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:45,639::task::595::TaskManager.Task::(_updateState) Task=`d5caadc1-96cc-49f7-b2af-84f4b63d7477`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:45,640::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:45,640::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:45,640::task::990::TaskManager.Task::(_decref) Task=`d5caadc1-96cc-49f7-b2af-84f4b63d7477`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:46,645::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:46,646::task::595::TaskManager.Task::(_updateState) Task=`c155e776-4d7f-4f05-81c5-efbb86246560`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:46,646::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:46,646::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:46,647::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:46,647::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:46,647::task::1185::TaskManager.Task::(prepare) Task=`c155e776-4d7f-4f05-81c5-efbb86246560`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:46,648::task::595::TaskManager.Task::(_updateState) Task=`c155e776-4d7f-4f05-81c5-efbb86246560`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:46,648::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:46,648::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:46,648::task::990::TaskManager.Task::(_decref) Task=`c155e776-4d7f-4f05-81c5-efbb86246560`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:47,654::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:47,655::task::595::TaskManager.Task::(_updateState) Task=`9f5f5b7a-1f80-4574-8528-d82d73b468f9`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:47,656::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:47,656::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:47,656::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:47,657::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:47,657::task::1185::TaskManager.Task::(prepare) Task=`9f5f5b7a-1f80-4574-8528-d82d73b468f9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:47,658::task::595::TaskManager.Task::(_updateState) Task=`9f5f5b7a-1f80-4574-8528-d82d73b468f9`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:47,658::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:47,658::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:47,659::task::990::TaskManager.Task::(_decref) Task=`9f5f5b7a-1f80-4574-8528-d82d73b468f9`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:48,664::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:48,664::task::595::TaskManager.Task::(_updateState) Task=`ffddc5c3-a8cf-4a55-8dcf-7f6cfc08818b`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:48,665::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:48,665::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:48,665::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:48,666::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:48,666::task::1185::TaskManager.Task::(prepare) Task=`ffddc5c3-a8cf-4a55-8dcf-7f6cfc08818b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:48,666::task::595::TaskManager.Task::(_updateState) Task=`ffddc5c3-a8cf-4a55-8dcf-7f6cfc08818b`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:48,667::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:48,667::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:48,667::task::990::TaskManager.Task::(_decref) Task=`ffddc5c3-a8cf-4a55-8dcf-7f6cfc08818b`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:49,672::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:49,672::task::595::TaskManager.Task::(_updateState) Task=`aafa764b-d395-4715-a338-35ed81f036d3`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:49,673::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:49,673::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:49,673::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:49,674::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:49,674::task::1185::TaskManager.Task::(prepare) Task=`aafa764b-d395-4715-a338-35ed81f036d3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:49,674::task::595::TaskManager.Task::(_updateState) Task=`aafa764b-d395-4715-a338-35ed81f036d3`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:49,674::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:49,675::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:49,675::task::990::TaskManager.Task::(_decref) Task=`aafa764b-d395-4715-a338-35ed81f036d3`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:49,905::task::595::TaskManager.Task::(_updateState) Task=`8698fbd6-9158-4654-be8b-40551af3d853`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:49,905::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:57:49,906::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000505038', 'lastCheck': '7.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000355257', 'lastCheck': '7.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000477959', 'lastCheck': '6.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '18.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000370907', 'lastCheck': '6.8', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:57:49,906::task::1185::TaskManager.Task::(prepare) Task=`8698fbd6-9158-4654-be8b-40551af3d853`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000505038', 'lastCheck': '7.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000355257', 'lastCheck': '7.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.7', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000477959', 'lastCheck': '6.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.8', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '18.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000370907', 'lastCheck': '6.8', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:57:49,907::task::595::TaskManager.Task::(_updateState) Task=`8698fbd6-9158-4654-be8b-40551af3d853`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:49,907::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:49,907::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:49,908::task::990::TaskManager.Task::(_decref) Task=`8698fbd6-9158-4654-be8b-40551af3d853`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:50,679::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:50,680::task::595::TaskManager.Task::(_updateState) Task=`c852c31d-b52c-4430-8218-7fa572ee4709`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:50,680::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:50,681::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:50,681::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:50,681::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:50,681::task::1185::TaskManager.Task::(prepare) Task=`c852c31d-b52c-4430-8218-7fa572ee4709`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:50,682::task::595::TaskManager.Task::(_updateState) Task=`c852c31d-b52c-4430-8218-7fa572ee4709`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:50,682::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:50,682::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:50,683::task::990::TaskManager.Task::(_decref) Task=`c852c31d-b52c-4430-8218-7fa572ee4709`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:51,687::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:51,688::task::595::TaskManager.Task::(_updateState) Task=`732a27a9-45d9-4119-93e5-04723929980a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:51,688::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:51,689::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:51,689::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:51,689::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:51,689::task::1185::TaskManager.Task::(prepare) Task=`732a27a9-45d9-4119-93e5-04723929980a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:51,690::task::595::TaskManager.Task::(_updateState) Task=`732a27a9-45d9-4119-93e5-04723929980a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:51,690::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:51,690::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:51,691::task::990::TaskManager.Task::(_decref) Task=`732a27a9-45d9-4119-93e5-04723929980a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:52,696::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:52,697::task::595::TaskManager.Task::(_updateState) Task=`b95479c8-f1aa-491e-a350-70150a9e56c2`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:52,698::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:52,698::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:52,699::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:52,699::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:52,699::task::1185::TaskManager.Task::(prepare) Task=`b95479c8-f1aa-491e-a350-70150a9e56c2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:52,700::task::595::TaskManager.Task::(_updateState) Task=`b95479c8-f1aa-491e-a350-70150a9e56c2`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:52,700::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:52,700::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:52,701::task::990::TaskManager.Task::(_decref) Task=`b95479c8-f1aa-491e-a350-70150a9e56c2`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:57:52,842::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:57:52,855::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000433634 s, 851 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:57:52,955::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:57:52,968::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000467098 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:57:53,096::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:57:53,119::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000480641 s, 1.6 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:57:53,153::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:57:53,164::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000400999 s, 1.9 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:57:53,706::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:53,707::task::595::TaskManager.Task::(_updateState) Task=`78686854-5150-444b-9591-d802b97dd454`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:53,708::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:53,708::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:53,708::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:53,709::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:53,709::task::1185::TaskManager.Task::(prepare) Task=`78686854-5150-444b-9591-d802b97dd454`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:53,710::task::595::TaskManager.Task::(_updateState) Task=`78686854-5150-444b-9591-d802b97dd454`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:53,710::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:53,710::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:53,711::task::990::TaskManager.Task::(_decref) Task=`78686854-5150-444b-9591-d802b97dd454`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:57:54,141::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:57:54,141::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:57:54,142::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:57:54,143::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:57:54,173::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:57:54,174::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:57:54,174::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:57:54,176::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:57:54,426::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:57:54,428::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-33::DEBUG::2014-04-11 22:57:54,455::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-39::ERROR::2014-04-11 22:57:54,457::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:57:54,458::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::WARNING::2014-04-11 22:57:54,459::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:57:54,460::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:57:54,468::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:57:54,469::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:57:54,716::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:54,717::task::595::TaskManager.Task::(_updateState) Task=`a710cdcc-d7cd-4b4c-889b-e47c2a89c58d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:54,718::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:54,718::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:54,718::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:54,718::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:54,719::task::1185::TaskManager.Task::(prepare) Task=`a710cdcc-d7cd-4b4c-889b-e47c2a89c58d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:54,719::task::595::TaskManager.Task::(_updateState) Task=`a710cdcc-d7cd-4b4c-889b-e47c2a89c58d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:54,719::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:54,720::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:54,720::task::990::TaskManager.Task::(_decref) Task=`a710cdcc-d7cd-4b4c-889b-e47c2a89c58d`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:55,725::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:55,725::task::595::TaskManager.Task::(_updateState) Task=`4c2b2522-21fe-4e7b-b54b-ef52178052ea`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:55,726::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:55,726::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:55,726::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:55,727::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:55,727::task::1185::TaskManager.Task::(prepare) Task=`4c2b2522-21fe-4e7b-b54b-ef52178052ea`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:55,727::task::595::TaskManager.Task::(_updateState) Task=`4c2b2522-21fe-4e7b-b54b-ef52178052ea`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:55,727::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:55,728::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:55,728::task::990::TaskManager.Task::(_decref) Task=`4c2b2522-21fe-4e7b-b54b-ef52178052ea`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:56,733::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:56,734::task::595::TaskManager.Task::(_updateState) Task=`6e302ab3-c074-48d6-9c55-3bc3c084d8e8`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:56,734::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:56,734::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:56,735::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:56,735::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:56,735::task::1185::TaskManager.Task::(prepare) Task=`6e302ab3-c074-48d6-9c55-3bc3c084d8e8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:56,735::task::595::TaskManager.Task::(_updateState) Task=`6e302ab3-c074-48d6-9c55-3bc3c084d8e8`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:56,736::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:56,736::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:56,736::task::990::TaskManager.Task::(_decref) Task=`6e302ab3-c074-48d6-9c55-3bc3c084d8e8`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:57,742::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:57,743::task::595::TaskManager.Task::(_updateState) Task=`8291e8d2-fc5f-4ffa-ac10-b0f43ba2bbe3`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:57,744::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:57,744::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:57,745::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:57,745::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:57,745::task::1185::TaskManager.Task::(prepare) Task=`8291e8d2-fc5f-4ffa-ac10-b0f43ba2bbe3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:57,746::task::595::TaskManager.Task::(_updateState) Task=`8291e8d2-fc5f-4ffa-ac10-b0f43ba2bbe3`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:57,746::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:57,746::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:57,747::task::990::TaskManager.Task::(_decref) Task=`8291e8d2-fc5f-4ffa-ac10-b0f43ba2bbe3`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:58,752::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:58,753::task::595::TaskManager.Task::(_updateState) Task=`3eee08cc-1b45-4c8d-ba84-b1e69472d38a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:58,753::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:58,753::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:58,754::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:58,754::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:58,754::task::1185::TaskManager.Task::(prepare) Task=`3eee08cc-1b45-4c8d-ba84-b1e69472d38a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:58,755::task::595::TaskManager.Task::(_updateState) Task=`3eee08cc-1b45-4c8d-ba84-b1e69472d38a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:58,755::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:58,755::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:58,756::task::990::TaskManager.Task::(_decref) Task=`3eee08cc-1b45-4c8d-ba84-b1e69472d38a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:57:59,760::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:57:59,761::task::595::TaskManager.Task::(_updateState) Task=`9bb4ad9a-e8f4-4353-98bf-1fe13514d013`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:57:59,762::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:57:59,762::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:57:59,762::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:57:59,763::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:59,763::task::1185::TaskManager.Task::(prepare) Task=`9bb4ad9a-e8f4-4353-98bf-1fe13514d013`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:57:59,763::task::595::TaskManager.Task::(_updateState) Task=`9bb4ad9a-e8f4-4353-98bf-1fe13514d013`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:57:59,764::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:57:59,764::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:57:59,764::task::990::TaskManager.Task::(_decref) Task=`9bb4ad9a-e8f4-4353-98bf-1fe13514d013`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:00,769::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:00,770::task::595::TaskManager.Task::(_updateState) Task=`f7f36f92-1431-46a9-aa56-1fb5bc41665d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:00,770::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:00,770::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:00,771::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:00,771::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:00,771::task::1185::TaskManager.Task::(prepare) Task=`f7f36f92-1431-46a9-aa56-1fb5bc41665d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:00,772::task::595::TaskManager.Task::(_updateState) Task=`f7f36f92-1431-46a9-aa56-1fb5bc41665d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:00,772::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:00,772::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:00,772::task::990::TaskManager.Task::(_decref) Task=`f7f36f92-1431-46a9-aa56-1fb5bc41665d`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:01,778::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:01,778::task::595::TaskManager.Task::(_updateState) Task=`bcb1d20c-2a73-4e9f-bb22-f080c535f984`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:01,779::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:01,779::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:01,779::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:01,780::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:01,780::task::1185::TaskManager.Task::(prepare) Task=`bcb1d20c-2a73-4e9f-bb22-f080c535f984`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:01,780::task::595::TaskManager.Task::(_updateState) Task=`bcb1d20c-2a73-4e9f-bb22-f080c535f984`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:01,781::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:01,781::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:01,781::task::990::TaskManager.Task::(_decref) Task=`bcb1d20c-2a73-4e9f-bb22-f080c535f984`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:02,790::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:02,791::task::595::TaskManager.Task::(_updateState) Task=`3f4bbd25-42d2-46de-9d33-016441c84529`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:02,792::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:02,792::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:02,792::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:02,793::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:02,793::task::1185::TaskManager.Task::(prepare) Task=`3f4bbd25-42d2-46de-9d33-016441c84529`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:02,794::task::595::TaskManager.Task::(_updateState) Task=`3f4bbd25-42d2-46de-9d33-016441c84529`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:02,794::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:02,794::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:02,795::task::990::TaskManager.Task::(_decref) Task=`3f4bbd25-42d2-46de-9d33-016441c84529`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:58:02,883::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:58:02,896::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000409128 s, 902 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:58:02,997::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:58:03,010::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000410639 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:58:03,156::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:58:03,168::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000456047 s, 1.7 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:58:03,199::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:58:03,211::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000305761 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:03,813::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:03,814::task::595::TaskManager.Task::(_updateState) Task=`80098a57-a998-466a-87fa-2e1f269a54ce`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:03,815::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:03,815::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:03,815::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:03,816::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:03,816::task::1185::TaskManager.Task::(prepare) Task=`80098a57-a998-466a-87fa-2e1f269a54ce`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:03,816::task::595::TaskManager.Task::(_updateState) Task=`80098a57-a998-466a-87fa-2e1f269a54ce`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:03,817::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:03,817::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:03,817::task::990::TaskManager.Task::(_decref) Task=`80098a57-a998-466a-87fa-2e1f269a54ce`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:58:04,459::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:58:04,460::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:58:04,460::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:58:04,461::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:58:04,471::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:58:04,472::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:58:04,473::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:58:04,475::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:58:04,732::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:58:04,734::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-33::DEBUG::2014-04-11 22:58:04,771::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-39::ERROR::2014-04-11 22:58:04,771::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:58:04,776::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::WARNING::2014-04-11 22:58:04,780::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:58:04,781::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:58:04,811::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:58:04,812::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:58:04,823::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:04,824::task::595::TaskManager.Task::(_updateState) Task=`ddb4d96e-0a37-4c15-b31e-3ad06f4536a5`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:04,825::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:04,825::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:04,825::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:04,826::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:04,826::task::1185::TaskManager.Task::(prepare) Task=`ddb4d96e-0a37-4c15-b31e-3ad06f4536a5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:04,827::task::595::TaskManager.Task::(_updateState) Task=`ddb4d96e-0a37-4c15-b31e-3ad06f4536a5`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:04,827::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:04,827::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:04,828::task::990::TaskManager.Task::(_decref) Task=`ddb4d96e-0a37-4c15-b31e-3ad06f4536a5`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:05,235::task::595::TaskManager.Task::(_updateState) Task=`6b29c5e0-c2e2-4dc9-a27a-b65dff86d4b5`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:05,236::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:58:05,237::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000410639', 'lastCheck': '2.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409128', 'lastCheck': '2.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000456047', 'lastCheck': '2.1', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '34.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000305761', 'lastCheck': '2.0', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:58:05,237::task::1185::TaskManager.Task::(prepare) Task=`6b29c5e0-c2e2-4dc9-a27a-b65dff86d4b5`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000410639', 'lastCheck': '2.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409128', 'lastCheck': '2.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000456047', 'lastCheck': '2.1', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '34.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000305761', 'lastCheck': '2.0', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:58:05,238::task::595::TaskManager.Task::(_updateState) Task=`6b29c5e0-c2e2-4dc9-a27a-b65dff86d4b5`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:05,238::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:05,238::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:05,239::task::990::TaskManager.Task::(_decref) Task=`6b29c5e0-c2e2-4dc9-a27a-b65dff86d4b5`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:05,833::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:05,833::task::595::TaskManager.Task::(_updateState) Task=`393f2aa8-2af6-4568-9882-fa49da6d187c`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:05,834::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:05,834::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:05,834::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:05,835::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:05,835::task::1185::TaskManager.Task::(prepare) Task=`393f2aa8-2af6-4568-9882-fa49da6d187c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:05,835::task::595::TaskManager.Task::(_updateState) Task=`393f2aa8-2af6-4568-9882-fa49da6d187c`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:05,836::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:05,836::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:05,836::task::990::TaskManager.Task::(_decref) Task=`393f2aa8-2af6-4568-9882-fa49da6d187c`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:06,841::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:06,842::task::595::TaskManager.Task::(_updateState) Task=`53102cd0-f81d-4f38-ad35-f8ba6d96052a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:06,842::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:06,843::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:06,843::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:06,843::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:06,844::task::1185::TaskManager.Task::(prepare) Task=`53102cd0-f81d-4f38-ad35-f8ba6d96052a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:06,844::task::595::TaskManager.Task::(_updateState) Task=`53102cd0-f81d-4f38-ad35-f8ba6d96052a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:06,844::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:06,844::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:06,845::task::990::TaskManager.Task::(_decref) Task=`53102cd0-f81d-4f38-ad35-f8ba6d96052a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:07,850::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:07,852::task::595::TaskManager.Task::(_updateState) Task=`7283f1d8-b650-4a86-ad29-2de242ee00ca`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:07,852::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:07,853::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:07,853::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:07,853::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:07,854::task::1185::TaskManager.Task::(prepare) Task=`7283f1d8-b650-4a86-ad29-2de242ee00ca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:07,854::task::595::TaskManager.Task::(_updateState) Task=`7283f1d8-b650-4a86-ad29-2de242ee00ca`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:07,855::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:07,855::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:07,855::task::990::TaskManager.Task::(_decref) Task=`7283f1d8-b650-4a86-ad29-2de242ee00ca`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:08,860::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:08,861::task::595::TaskManager.Task::(_updateState) Task=`cd1f4eff-de21-4ce0-a8db-4c319bb3eabf`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:08,862::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:08,862::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:08,862::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:08,863::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:08,863::task::1185::TaskManager.Task::(prepare) Task=`cd1f4eff-de21-4ce0-a8db-4c319bb3eabf`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:08,863::task::595::TaskManager.Task::(_updateState) Task=`cd1f4eff-de21-4ce0-a8db-4c319bb3eabf`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:08,864::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:08,864::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:08,864::task::990::TaskManager.Task::(_decref) Task=`cd1f4eff-de21-4ce0-a8db-4c319bb3eabf`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:09,869::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:09,869::task::595::TaskManager.Task::(_updateState) Task=`c5eca62d-fa5c-4998-8b05-1d8734cfe8cc`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:09,870::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:09,870::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:09,870::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:09,871::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:09,871::task::1185::TaskManager.Task::(prepare) Task=`c5eca62d-fa5c-4998-8b05-1d8734cfe8cc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:09,871::task::595::TaskManager.Task::(_updateState) Task=`c5eca62d-fa5c-4998-8b05-1d8734cfe8cc`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:09,872::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:09,872::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:09,872::task::990::TaskManager.Task::(_decref) Task=`c5eca62d-fa5c-4998-8b05-1d8734cfe8cc`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:10,877::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:10,877::task::595::TaskManager.Task::(_updateState) Task=`f7930e3a-a85e-474f-90d2-2a1f5e14fc73`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:10,878::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:10,878::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:10,878::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:10,879::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:10,879::task::1185::TaskManager.Task::(prepare) Task=`f7930e3a-a85e-474f-90d2-2a1f5e14fc73`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:10,879::task::595::TaskManager.Task::(_updateState) Task=`f7930e3a-a85e-474f-90d2-2a1f5e14fc73`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:10,879::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:10,880::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:10,880::task::990::TaskManager.Task::(_decref) Task=`f7930e3a-a85e-474f-90d2-2a1f5e14fc73`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:11,885::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:11,885::task::595::TaskManager.Task::(_updateState) Task=`63cd38c0-c713-4e74-98fb-f623e1f9ab17`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:11,886::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:11,886::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:11,886::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:11,886::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:11,887::task::1185::TaskManager.Task::(prepare) Task=`63cd38c0-c713-4e74-98fb-f623e1f9ab17`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:11,887::task::595::TaskManager.Task::(_updateState) Task=`63cd38c0-c713-4e74-98fb-f623e1f9ab17`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:11,887::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:11,888::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:11,888::task::990::TaskManager.Task::(_decref) Task=`63cd38c0-c713-4e74-98fb-f623e1f9ab17`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:12,894::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:12,895::task::595::TaskManager.Task::(_updateState) Task=`c14d45b5-9b12-45f5-b375-417a72bcaee1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:12,895::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:12,896::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:12,896::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:12,896::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:12,897::task::1185::TaskManager.Task::(prepare) Task=`c14d45b5-9b12-45f5-b375-417a72bcaee1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:12,897::task::595::TaskManager.Task::(_updateState) Task=`c14d45b5-9b12-45f5-b375-417a72bcaee1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:12,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:12,898::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:12,898::task::990::TaskManager.Task::(_decref) Task=`c14d45b5-9b12-45f5-b375-417a72bcaee1`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:58:12,929::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:58:12,942::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.00038346 s, 962 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:58:13,038::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:58:13,051::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000477924 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:58:13,200::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:58:13,212::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000459218 s, 1.6 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:58:13,241::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:58:13,252::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000364313 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:13,909::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:13,910::task::595::TaskManager.Task::(_updateState) Task=`970396ac-40c5-4d47-9de9-198c8fbcf2ae`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:13,911::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:13,911::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:13,911::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:13,912::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:13,912::task::1185::TaskManager.Task::(prepare) Task=`970396ac-40c5-4d47-9de9-198c8fbcf2ae`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:13,913::task::595::TaskManager.Task::(_updateState) Task=`970396ac-40c5-4d47-9de9-198c8fbcf2ae`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:13,913::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:13,913::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:13,914::task::990::TaskManager.Task::(_decref) Task=`970396ac-40c5-4d47-9de9-198c8fbcf2ae`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:58:14,781::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:58:14,781::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:58:14,781::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:58:14,783::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:58:14,813::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:58:14,814::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:58:14,815::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:58:14,816::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 22:58:14,920::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:14,922::task::595::TaskManager.Task::(_updateState) Task=`aa551611-7b34-41fe-9d2a-1089612b6da1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:14,922::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:14,923::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:14,923::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:14,923::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:14,924::task::1185::TaskManager.Task::(prepare) Task=`aa551611-7b34-41fe-9d2a-1089612b6da1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:14,924::task::595::TaskManager.Task::(_updateState) Task=`aa551611-7b34-41fe-9d2a-1089612b6da1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:14,924::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:14,925::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:14,925::task::990::TaskManager.Task::(_decref) Task=`aa551611-7b34-41fe-9d2a-1089612b6da1`::ref 0 aborting False >Thread-39::DEBUG::2014-04-11 22:58:15,051::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:58:15,053::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:58:15,085::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:58:15,086::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:58:15,088::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:58:15,090::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:58:15,090::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:58:15,119::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:58:15,120::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:58:15,931::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:15,931::task::595::TaskManager.Task::(_updateState) Task=`a8abc817-f46f-4730-a0ee-c7674b438b90`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:15,932::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:15,932::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:15,932::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:15,933::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:15,933::task::1185::TaskManager.Task::(prepare) Task=`a8abc817-f46f-4730-a0ee-c7674b438b90`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:15,933::task::595::TaskManager.Task::(_updateState) Task=`a8abc817-f46f-4730-a0ee-c7674b438b90`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:15,934::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:15,934::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:15,934::task::990::TaskManager.Task::(_decref) Task=`a8abc817-f46f-4730-a0ee-c7674b438b90`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:16,939::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:16,940::task::595::TaskManager.Task::(_updateState) Task=`0499996a-6da6-44d4-b577-022fd8a8a25e`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:16,940::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:16,940::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:16,941::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:16,941::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:16,941::task::1185::TaskManager.Task::(prepare) Task=`0499996a-6da6-44d4-b577-022fd8a8a25e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:16,942::task::595::TaskManager.Task::(_updateState) Task=`0499996a-6da6-44d4-b577-022fd8a8a25e`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:16,942::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:16,943::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:16,943::task::990::TaskManager.Task::(_decref) Task=`0499996a-6da6-44d4-b577-022fd8a8a25e`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:17,948::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:17,950::task::595::TaskManager.Task::(_updateState) Task=`bd45977b-b3f4-497a-89af-a5bbd4ff10cc`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:17,950::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:17,950::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:17,951::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:17,951::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:17,952::task::1185::TaskManager.Task::(prepare) Task=`bd45977b-b3f4-497a-89af-a5bbd4ff10cc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:17,952::task::595::TaskManager.Task::(_updateState) Task=`bd45977b-b3f4-497a-89af-a5bbd4ff10cc`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:17,952::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:17,953::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:17,953::task::990::TaskManager.Task::(_decref) Task=`bd45977b-b3f4-497a-89af-a5bbd4ff10cc`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:18,958::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:18,959::task::595::TaskManager.Task::(_updateState) Task=`0a1fb5d6-7b11-44f3-8d9a-9229e657db5d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:18,959::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:18,959::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:18,960::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:18,960::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:18,960::task::1185::TaskManager.Task::(prepare) Task=`0a1fb5d6-7b11-44f3-8d9a-9229e657db5d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:18,960::task::595::TaskManager.Task::(_updateState) Task=`0a1fb5d6-7b11-44f3-8d9a-9229e657db5d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:18,961::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:18,961::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:18,961::task::990::TaskManager.Task::(_decref) Task=`0a1fb5d6-7b11-44f3-8d9a-9229e657db5d`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:19,966::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:19,982::task::595::TaskManager.Task::(_updateState) Task=`96bc4fb1-ab05-43b7-97dc-8bd112137c51`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:19,983::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:19,983::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:19,983::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:19,984::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:19,984::task::1185::TaskManager.Task::(prepare) Task=`96bc4fb1-ab05-43b7-97dc-8bd112137c51`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:19,985::task::595::TaskManager.Task::(_updateState) Task=`96bc4fb1-ab05-43b7-97dc-8bd112137c51`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:19,985::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:19,985::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:19,986::task::990::TaskManager.Task::(_decref) Task=`96bc4fb1-ab05-43b7-97dc-8bd112137c51`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:20,556::task::595::TaskManager.Task::(_updateState) Task=`4762bfee-b14c-4a98-82be-d191002d4b55`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:20,556::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:58:20,557::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000477924', 'lastCheck': '7.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00038346', 'lastCheck': '7.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000459218', 'lastCheck': '7.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '49.5', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000364313', 'lastCheck': '7.3', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:58:20,557::task::1185::TaskManager.Task::(prepare) Task=`4762bfee-b14c-4a98-82be-d191002d4b55`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000477924', 'lastCheck': '7.5', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00038346', 'lastCheck': '7.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.4', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000459218', 'lastCheck': '7.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '49.5', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000364313', 'lastCheck': '7.3', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:58:20,558::task::595::TaskManager.Task::(_updateState) Task=`4762bfee-b14c-4a98-82be-d191002d4b55`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:20,558::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:20,558::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:20,559::task::990::TaskManager.Task::(_decref) Task=`4762bfee-b14c-4a98-82be-d191002d4b55`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:20,991::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:20,991::task::595::TaskManager.Task::(_updateState) Task=`f4034e85-51d0-427a-8f5e-efd2414f5538`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:20,992::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:20,992::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:20,992::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:20,993::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:20,993::task::1185::TaskManager.Task::(prepare) Task=`f4034e85-51d0-427a-8f5e-efd2414f5538`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:20,993::task::595::TaskManager.Task::(_updateState) Task=`f4034e85-51d0-427a-8f5e-efd2414f5538`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:20,994::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:20,994::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:20,994::task::990::TaskManager.Task::(_decref) Task=`f4034e85-51d0-427a-8f5e-efd2414f5538`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:21,999::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:22,000::task::595::TaskManager.Task::(_updateState) Task=`bfd7f20e-0d30-414f-bff9-13f57c41f89b`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:22,000::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:22,000::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:22,000::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:22,001::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:22,001::task::1185::TaskManager.Task::(prepare) Task=`bfd7f20e-0d30-414f-bff9-13f57c41f89b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:22,001::task::595::TaskManager.Task::(_updateState) Task=`bfd7f20e-0d30-414f-bff9-13f57c41f89b`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:22,002::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:22,002::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:22,002::task::990::TaskManager.Task::(_decref) Task=`bfd7f20e-0d30-414f-bff9-13f57c41f89b`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:58:22,971::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:58:22,989::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000313431 s, 1.2 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:23,008::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:23,009::task::595::TaskManager.Task::(_updateState) Task=`af1907a0-7c03-493d-b0b3-60d9bb633a71`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:23,010::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:23,010::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:23,010::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:23,011::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:23,011::task::1185::TaskManager.Task::(prepare) Task=`af1907a0-7c03-493d-b0b3-60d9bb633a71`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:23,012::task::595::TaskManager.Task::(_updateState) Task=`af1907a0-7c03-493d-b0b3-60d9bb633a71`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:23,012::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:23,012::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:23,013::task::990::TaskManager.Task::(_decref) Task=`af1907a0-7c03-493d-b0b3-60d9bb633a71`::ref 0 aborting False >Thread-22::DEBUG::2014-04-11 22:58:23,079::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:58:23,093::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000454332 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:58:23,239::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:58:23,251::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000441898 s, 1.7 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:58:23,282::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:58:23,293::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000285471 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:24,018::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:24,018::task::595::TaskManager.Task::(_updateState) Task=`8c4fee41-f74b-4ec7-bc70-6bb49c552587`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:24,019::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:24,019::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:24,020::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:24,020::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:24,020::task::1185::TaskManager.Task::(prepare) Task=`8c4fee41-f74b-4ec7-bc70-6bb49c552587`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:24,021::task::595::TaskManager.Task::(_updateState) Task=`8c4fee41-f74b-4ec7-bc70-6bb49c552587`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:24,021::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:24,021::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:24,022::task::990::TaskManager.Task::(_decref) Task=`8c4fee41-f74b-4ec7-bc70-6bb49c552587`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:25,026::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:25,046::task::595::TaskManager.Task::(_updateState) Task=`9c683637-cc7c-450b-80c4-ae5818ba9f0d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:25,047::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:25,047::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:25,047::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:25,048::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:25,048::task::1185::TaskManager.Task::(prepare) Task=`9c683637-cc7c-450b-80c4-ae5818ba9f0d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:25,048::task::595::TaskManager.Task::(_updateState) Task=`9c683637-cc7c-450b-80c4-ae5818ba9f0d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:25,048::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:25,049::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:25,049::task::990::TaskManager.Task::(_decref) Task=`9c683637-cc7c-450b-80c4-ae5818ba9f0d`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:58:25,087::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:58:25,087::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:58:25,087::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:58:25,088::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:58:25,121::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:58:25,122::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:58:25,122::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:58:25,123::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:58:25,356::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:58:25,358::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:58:25,392::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:58:25,393::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:58:25,396::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:58:25,398::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:58:25,398::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:58:25,427::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:58:25,428::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:58:26,054::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:26,055::task::595::TaskManager.Task::(_updateState) Task=`0134ab71-3642-4443-8d49-83ea61d0c684`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:26,056::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:26,056::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:26,057::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:26,057::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:26,057::task::1185::TaskManager.Task::(prepare) Task=`0134ab71-3642-4443-8d49-83ea61d0c684`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:26,058::task::595::TaskManager.Task::(_updateState) Task=`0134ab71-3642-4443-8d49-83ea61d0c684`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:26,058::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:26,058::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:26,059::task::990::TaskManager.Task::(_decref) Task=`0134ab71-3642-4443-8d49-83ea61d0c684`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:27,064::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:27,065::task::595::TaskManager.Task::(_updateState) Task=`ab748b99-7754-42cf-94a6-3489a07aa47f`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:27,065::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:27,065::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:27,066::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:27,066::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:27,066::task::1185::TaskManager.Task::(prepare) Task=`ab748b99-7754-42cf-94a6-3489a07aa47f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:27,067::task::595::TaskManager.Task::(_updateState) Task=`ab748b99-7754-42cf-94a6-3489a07aa47f`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:27,067::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:27,067::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:27,068::task::990::TaskManager.Task::(_decref) Task=`ab748b99-7754-42cf-94a6-3489a07aa47f`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:28,073::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:28,074::task::595::TaskManager.Task::(_updateState) Task=`56ca247d-0045-4245-b91c-c721bbbd0310`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:28,075::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:28,075::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:28,076::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:28,076::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:28,076::task::1185::TaskManager.Task::(prepare) Task=`56ca247d-0045-4245-b91c-c721bbbd0310`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:28,077::task::595::TaskManager.Task::(_updateState) Task=`56ca247d-0045-4245-b91c-c721bbbd0310`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:28,077::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:28,078::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:28,078::task::990::TaskManager.Task::(_decref) Task=`56ca247d-0045-4245-b91c-c721bbbd0310`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:29,084::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:29,096::task::595::TaskManager.Task::(_updateState) Task=`4132d974-2466-4e2b-a611-cdd1cc5be908`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:29,096::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:29,097::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:29,097::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:29,097::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:29,098::task::1185::TaskManager.Task::(prepare) Task=`4132d974-2466-4e2b-a611-cdd1cc5be908`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:29,098::task::595::TaskManager.Task::(_updateState) Task=`4132d974-2466-4e2b-a611-cdd1cc5be908`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:29,098::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:29,099::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:29,099::task::990::TaskManager.Task::(_decref) Task=`4132d974-2466-4e2b-a611-cdd1cc5be908`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:30,105::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:30,106::task::595::TaskManager.Task::(_updateState) Task=`bf105be5-eb52-4976-bdd2-005bc9b3c705`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:30,106::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:30,106::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:30,107::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:30,107::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:30,107::task::1185::TaskManager.Task::(prepare) Task=`bf105be5-eb52-4976-bdd2-005bc9b3c705`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:30,108::task::595::TaskManager.Task::(_updateState) Task=`bf105be5-eb52-4976-bdd2-005bc9b3c705`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:30,108::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:30,108::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:30,109::task::990::TaskManager.Task::(_decref) Task=`bf105be5-eb52-4976-bdd2-005bc9b3c705`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:31,114::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:31,115::task::595::TaskManager.Task::(_updateState) Task=`3df757b9-9e35-4ba8-8606-23683df775ee`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:31,115::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:31,115::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:31,116::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:31,116::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:31,116::task::1185::TaskManager.Task::(prepare) Task=`3df757b9-9e35-4ba8-8606-23683df775ee`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:31,117::task::595::TaskManager.Task::(_updateState) Task=`3df757b9-9e35-4ba8-8606-23683df775ee`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:31,117::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:31,117::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:31,117::task::990::TaskManager.Task::(_decref) Task=`3df757b9-9e35-4ba8-8606-23683df775ee`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:32,122::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:32,122::task::595::TaskManager.Task::(_updateState) Task=`3a634006-72fb-4a2f-bc18-ad56ec3f6ff5`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:32,123::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:32,123::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:32,123::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:32,124::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:32,124::task::1185::TaskManager.Task::(prepare) Task=`3a634006-72fb-4a2f-bc18-ad56ec3f6ff5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:32,124::task::595::TaskManager.Task::(_updateState) Task=`3a634006-72fb-4a2f-bc18-ad56ec3f6ff5`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:32,124::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:32,125::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:32,125::task::990::TaskManager.Task::(_decref) Task=`3a634006-72fb-4a2f-bc18-ad56ec3f6ff5`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:58:33,020::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:58:33,033::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000346881 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:58:33,122::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-14::DEBUG::2014-04-11 22:58:33,137::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:33,142::task::595::TaskManager.Task::(_updateState) Task=`da7044f7-623c-4619-b4d5-6c113aa3f079`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:33,145::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:33,147::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:33,149::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:33,151::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:33,153::task::1185::TaskManager.Task::(prepare) Task=`da7044f7-623c-4619-b4d5-6c113aa3f079`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:33,156::task::595::TaskManager.Task::(_updateState) Task=`da7044f7-623c-4619-b4d5-6c113aa3f079`::moving from state preparing -> state finished >Thread-22::DEBUG::2014-04-11 22:58:33,157::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000510101 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:33,158::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:33,161::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:33,162::task::990::TaskManager.Task::(_decref) Task=`da7044f7-623c-4619-b4d5-6c113aa3f079`::ref 0 aborting False >Thread-29::DEBUG::2014-04-11 22:58:33,279::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:58:33,291::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000472078 s, 1.6 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:58:33,323::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:58:33,335::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000320522 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:34,172::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:34,173::task::595::TaskManager.Task::(_updateState) Task=`cc69ed9d-d654-4c11-a693-d3a5a6cc1af8`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:34,173::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:34,174::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:34,174::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:34,174::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:34,174::task::1185::TaskManager.Task::(prepare) Task=`cc69ed9d-d654-4c11-a693-d3a5a6cc1af8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:34,175::task::595::TaskManager.Task::(_updateState) Task=`cc69ed9d-d654-4c11-a693-d3a5a6cc1af8`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:34,175::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:34,175::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:34,176::task::990::TaskManager.Task::(_decref) Task=`cc69ed9d-d654-4c11-a693-d3a5a6cc1af8`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:35,180::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:35,181::task::595::TaskManager.Task::(_updateState) Task=`2edb7202-8919-475d-8b48-df18f244aef0`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:35,181::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:35,181::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:35,182::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:35,182::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:35,182::task::1185::TaskManager.Task::(prepare) Task=`2edb7202-8919-475d-8b48-df18f244aef0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:35,183::task::595::TaskManager.Task::(_updateState) Task=`2edb7202-8919-475d-8b48-df18f244aef0`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:35,183::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:35,183::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:35,183::task::990::TaskManager.Task::(_decref) Task=`2edb7202-8919-475d-8b48-df18f244aef0`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:58:35,394::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:58:35,394::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:58:35,395::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:58:35,395::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:58:35,429::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:58:35,430::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:58:35,430::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:58:35,431::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:58:35,666::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:58:35,668::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:58:35,704::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:58:35,706::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:58:35,712::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:58:35,714::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:58:35,714::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:58:35,745::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:58:35,746::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:58:35,879::task::595::TaskManager.Task::(_updateState) Task=`db6372b5-758e-4df7-83da-b53d8b80cd74`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:35,879::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:58:35,880::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000510101', 'lastCheck': '2.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346881', 'lastCheck': '2.8', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000472078', 'lastCheck': '2.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '64.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000320522', 'lastCheck': '2.5', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:58:35,881::task::1185::TaskManager.Task::(prepare) Task=`db6372b5-758e-4df7-83da-b53d8b80cd74`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000510101', 'lastCheck': '2.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346881', 'lastCheck': '2.8', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000472078', 'lastCheck': '2.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '64.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000320522', 'lastCheck': '2.5', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:58:35,881::task::595::TaskManager.Task::(_updateState) Task=`db6372b5-758e-4df7-83da-b53d8b80cd74`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:35,881::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:35,882::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:35,882::task::990::TaskManager.Task::(_decref) Task=`db6372b5-758e-4df7-83da-b53d8b80cd74`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:36,188::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:36,189::task::595::TaskManager.Task::(_updateState) Task=`c7158d77-ee39-4bd7-89f7-96c7d5e201b2`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:36,189::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:36,189::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:36,190::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:36,190::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:36,190::task::1185::TaskManager.Task::(prepare) Task=`c7158d77-ee39-4bd7-89f7-96c7d5e201b2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:36,191::task::595::TaskManager.Task::(_updateState) Task=`c7158d77-ee39-4bd7-89f7-96c7d5e201b2`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:36,191::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:36,191::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:36,192::task::990::TaskManager.Task::(_decref) Task=`c7158d77-ee39-4bd7-89f7-96c7d5e201b2`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:37,196::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:37,197::task::595::TaskManager.Task::(_updateState) Task=`3e5da922-0b68-4748-829b-a1cd5e7c1ae2`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:37,197::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:37,198::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:37,198::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:37,198::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:37,198::task::1185::TaskManager.Task::(prepare) Task=`3e5da922-0b68-4748-829b-a1cd5e7c1ae2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:37,199::task::595::TaskManager.Task::(_updateState) Task=`3e5da922-0b68-4748-829b-a1cd5e7c1ae2`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:37,199::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:37,199::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:37,200::task::990::TaskManager.Task::(_decref) Task=`3e5da922-0b68-4748-829b-a1cd5e7c1ae2`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:38,205::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:38,206::task::595::TaskManager.Task::(_updateState) Task=`6f829d28-f7b2-429c-9b06-79518353d352`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:38,207::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:38,207::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:38,208::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:38,208::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:38,208::task::1185::TaskManager.Task::(prepare) Task=`6f829d28-f7b2-429c-9b06-79518353d352`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:38,209::task::595::TaskManager.Task::(_updateState) Task=`6f829d28-f7b2-429c-9b06-79518353d352`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:38,209::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:38,209::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:38,210::task::990::TaskManager.Task::(_decref) Task=`6f829d28-f7b2-429c-9b06-79518353d352`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:39,215::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:39,216::task::595::TaskManager.Task::(_updateState) Task=`ba2f3462-a98a-401a-b379-a33a6923ff49`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:39,216::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:39,216::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:39,217::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:39,217::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:39,217::task::1185::TaskManager.Task::(prepare) Task=`ba2f3462-a98a-401a-b379-a33a6923ff49`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:39,218::task::595::TaskManager.Task::(_updateState) Task=`ba2f3462-a98a-401a-b379-a33a6923ff49`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:39,218::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:39,218::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:39,219::task::990::TaskManager.Task::(_decref) Task=`ba2f3462-a98a-401a-b379-a33a6923ff49`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:40,223::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:40,224::task::595::TaskManager.Task::(_updateState) Task=`323d7ae6-6411-48ac-99cb-2e71a04a975a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:40,224::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:40,224::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:40,225::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:40,225::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:40,225::task::1185::TaskManager.Task::(prepare) Task=`323d7ae6-6411-48ac-99cb-2e71a04a975a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:40,226::task::595::TaskManager.Task::(_updateState) Task=`323d7ae6-6411-48ac-99cb-2e71a04a975a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:40,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:40,226::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:40,226::task::990::TaskManager.Task::(_decref) Task=`323d7ae6-6411-48ac-99cb-2e71a04a975a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:41,231::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:41,232::task::595::TaskManager.Task::(_updateState) Task=`4200bb4c-1553-4cee-bd93-3f409033c609`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:41,233::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:41,233::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:41,233::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:41,233::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:41,234::task::1185::TaskManager.Task::(prepare) Task=`4200bb4c-1553-4cee-bd93-3f409033c609`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:41,234::task::595::TaskManager.Task::(_updateState) Task=`4200bb4c-1553-4cee-bd93-3f409033c609`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:41,234::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:41,235::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:41,235::task::990::TaskManager.Task::(_decref) Task=`4200bb4c-1553-4cee-bd93-3f409033c609`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:42,239::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:42,240::task::595::TaskManager.Task::(_updateState) Task=`f5d45540-bb27-4559-8d30-08259c974650`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:42,241::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:42,241::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:42,241::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:42,241::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:42,242::task::1185::TaskManager.Task::(prepare) Task=`f5d45540-bb27-4559-8d30-08259c974650`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:42,242::task::595::TaskManager.Task::(_updateState) Task=`f5d45540-bb27-4559-8d30-08259c974650`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:42,242::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:42,243::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:42,243::task::990::TaskManager.Task::(_decref) Task=`f5d45540-bb27-4559-8d30-08259c974650`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:58:43,062::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:58:43,075::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000346237 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:58:43,193::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:58:43,206::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.00049129 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:43,248::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:43,249::task::595::TaskManager.Task::(_updateState) Task=`23d62849-b7bc-4c74-8279-49a3e9e28eda`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:43,250::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:43,250::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:43,250::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:43,251::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:43,251::task::1185::TaskManager.Task::(prepare) Task=`23d62849-b7bc-4c74-8279-49a3e9e28eda`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:43,251::task::595::TaskManager.Task::(_updateState) Task=`23d62849-b7bc-4c74-8279-49a3e9e28eda`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:43,252::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:43,252::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:43,252::task::990::TaskManager.Task::(_decref) Task=`23d62849-b7bc-4c74-8279-49a3e9e28eda`::ref 0 aborting False >Thread-29::DEBUG::2014-04-11 22:58:43,319::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:58:43,331::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000465492 s, 1.6 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:58:43,364::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:58:43,376::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000346317 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:44,258::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:44,259::task::595::TaskManager.Task::(_updateState) Task=`4cd0c327-5a64-473d-b8d1-4b32c111f17c`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:44,259::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:44,260::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:44,260::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:44,260::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:44,261::task::1185::TaskManager.Task::(prepare) Task=`4cd0c327-5a64-473d-b8d1-4b32c111f17c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:44,261::task::595::TaskManager.Task::(_updateState) Task=`4cd0c327-5a64-473d-b8d1-4b32c111f17c`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:44,261::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:44,262::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:44,262::task::990::TaskManager.Task::(_decref) Task=`4cd0c327-5a64-473d-b8d1-4b32c111f17c`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:45,267::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:45,267::task::595::TaskManager.Task::(_updateState) Task=`105d6394-83db-445f-be58-2f47e5353350`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:45,268::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:45,268::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:45,269::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:45,269::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:45,269::task::1185::TaskManager.Task::(prepare) Task=`105d6394-83db-445f-be58-2f47e5353350`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:45,270::task::595::TaskManager.Task::(_updateState) Task=`105d6394-83db-445f-be58-2f47e5353350`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:45,270::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:45,270::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:45,271::task::990::TaskManager.Task::(_decref) Task=`105d6394-83db-445f-be58-2f47e5353350`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:58:45,712::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:58:45,712::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:58:45,713::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:58:45,714::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:58:45,747::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:58:45,748::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:58:45,748::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:58:45,749::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:58:45,981::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:58:45,983::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:58:46,021::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:58:46,022::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:58:46,046::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:58:46,048::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:58:46,049::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:58:46,077::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:58:46,078::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:58:46,276::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:46,277::task::595::TaskManager.Task::(_updateState) Task=`ef181863-c764-4b28-9bb7-cbdd69d30a9c`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:46,278::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:46,278::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:46,278::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:46,279::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:46,279::task::1185::TaskManager.Task::(prepare) Task=`ef181863-c764-4b28-9bb7-cbdd69d30a9c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:46,279::task::595::TaskManager.Task::(_updateState) Task=`ef181863-c764-4b28-9bb7-cbdd69d30a9c`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:46,280::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:46,280::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:46,280::task::990::TaskManager.Task::(_decref) Task=`ef181863-c764-4b28-9bb7-cbdd69d30a9c`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:47,285::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:47,286::task::595::TaskManager.Task::(_updateState) Task=`d217fba9-3564-4248-b266-c2eb7bd74cc1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:47,287::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:47,287::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:47,287::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:47,288::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:47,288::task::1185::TaskManager.Task::(prepare) Task=`d217fba9-3564-4248-b266-c2eb7bd74cc1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:47,288::task::595::TaskManager.Task::(_updateState) Task=`d217fba9-3564-4248-b266-c2eb7bd74cc1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:47,289::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:47,289::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:47,289::task::990::TaskManager.Task::(_decref) Task=`d217fba9-3564-4248-b266-c2eb7bd74cc1`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:48,294::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:48,295::task::595::TaskManager.Task::(_updateState) Task=`fe290b89-9977-4d14-81b7-03d66a490d58`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:48,296::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:48,296::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:48,296::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:48,297::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:48,297::task::1185::TaskManager.Task::(prepare) Task=`fe290b89-9977-4d14-81b7-03d66a490d58`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:48,297::task::595::TaskManager.Task::(_updateState) Task=`fe290b89-9977-4d14-81b7-03d66a490d58`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:48,298::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:48,298::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:48,298::task::990::TaskManager.Task::(_decref) Task=`fe290b89-9977-4d14-81b7-03d66a490d58`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:49,303::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:49,304::task::595::TaskManager.Task::(_updateState) Task=`5f40abae-cea1-4b1d-9c48-d34f6f0933ec`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:49,304::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:49,304::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:49,304::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:49,305::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:49,305::task::1185::TaskManager.Task::(prepare) Task=`5f40abae-cea1-4b1d-9c48-d34f6f0933ec`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:49,305::task::595::TaskManager.Task::(_updateState) Task=`5f40abae-cea1-4b1d-9c48-d34f6f0933ec`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:49,306::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:49,306::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:49,306::task::990::TaskManager.Task::(_decref) Task=`5f40abae-cea1-4b1d-9c48-d34f6f0933ec`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:50,311::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:50,311::task::595::TaskManager.Task::(_updateState) Task=`9fe8e2c0-fb33-4eb2-b154-9f677faa21d0`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:50,312::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:50,312::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:50,312::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:50,312::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:50,313::task::1185::TaskManager.Task::(prepare) Task=`9fe8e2c0-fb33-4eb2-b154-9f677faa21d0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:50,313::task::595::TaskManager.Task::(_updateState) Task=`9fe8e2c0-fb33-4eb2-b154-9f677faa21d0`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:50,313::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:50,314::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:50,314::task::990::TaskManager.Task::(_decref) Task=`9fe8e2c0-fb33-4eb2-b154-9f677faa21d0`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:51,183::task::595::TaskManager.Task::(_updateState) Task=`5caa8434-eb80-4718-ba19-f75004553fb6`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:51,183::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:58:51,184::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00049129', 'lastCheck': '8.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346237', 'lastCheck': '8.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000465492', 'lastCheck': '7.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '80.1', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000346317', 'lastCheck': '7.8', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:58:51,184::task::1185::TaskManager.Task::(prepare) Task=`5caa8434-eb80-4718-ba19-f75004553fb6`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00049129', 'lastCheck': '8.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346237', 'lastCheck': '8.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000465492', 'lastCheck': '7.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '80.1', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000346317', 'lastCheck': '7.8', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:58:51,185::task::595::TaskManager.Task::(_updateState) Task=`5caa8434-eb80-4718-ba19-f75004553fb6`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:51,185::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:51,185::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:51,186::task::990::TaskManager.Task::(_decref) Task=`5caa8434-eb80-4718-ba19-f75004553fb6`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:51,318::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:51,319::task::595::TaskManager.Task::(_updateState) Task=`3341692a-6936-429c-a8f9-ea476b959f96`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:51,319::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:51,319::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:51,320::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:51,320::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:51,320::task::1185::TaskManager.Task::(prepare) Task=`3341692a-6936-429c-a8f9-ea476b959f96`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:51,321::task::595::TaskManager.Task::(_updateState) Task=`3341692a-6936-429c-a8f9-ea476b959f96`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:51,321::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:51,321::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:51,321::task::990::TaskManager.Task::(_decref) Task=`3341692a-6936-429c-a8f9-ea476b959f96`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:52,326::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:52,326::task::595::TaskManager.Task::(_updateState) Task=`f14785b4-9af1-4d4e-b104-21e567cc7e6b`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:52,327::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:52,327::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:52,327::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:52,328::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:52,328::task::1185::TaskManager.Task::(prepare) Task=`f14785b4-9af1-4d4e-b104-21e567cc7e6b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:52,328::task::595::TaskManager.Task::(_updateState) Task=`f14785b4-9af1-4d4e-b104-21e567cc7e6b`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:52,328::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:52,329::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:52,329::task::990::TaskManager.Task::(_decref) Task=`f14785b4-9af1-4d4e-b104-21e567cc7e6b`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:58:53,103::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:58:53,116::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000416235 s, 887 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:58:53,234::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:58:53,248::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.0004171 s, 1.7 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:53,337::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:53,343::task::595::TaskManager.Task::(_updateState) Task=`5ca7d2ea-0874-4d52-88bf-f7920bd0e1b6`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:53,348::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:53,351::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:53,354::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:53,358::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:53,361::task::1185::TaskManager.Task::(prepare) Task=`5ca7d2ea-0874-4d52-88bf-f7920bd0e1b6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:53,365::task::595::TaskManager.Task::(_updateState) Task=`5ca7d2ea-0874-4d52-88bf-f7920bd0e1b6`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:53,369::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:53,372::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:53,376::task::990::TaskManager.Task::(_decref) Task=`5ca7d2ea-0874-4d52-88bf-f7920bd0e1b6`::ref 0 aborting False >Thread-29::DEBUG::2014-04-11 22:58:53,549::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:58:53,583::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000389616 s, 1.9 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:58:53,595::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:58:53,607::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000351646 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:58:54,396::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:54,397::task::595::TaskManager.Task::(_updateState) Task=`8ebd4e2a-aa9f-45ee-b0ee-655df4c55b10`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:54,397::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:54,398::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:54,398::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:54,398::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:54,399::task::1185::TaskManager.Task::(prepare) Task=`8ebd4e2a-aa9f-45ee-b0ee-655df4c55b10`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:54,399::task::595::TaskManager.Task::(_updateState) Task=`8ebd4e2a-aa9f-45ee-b0ee-655df4c55b10`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:54,399::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:54,400::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:54,400::task::990::TaskManager.Task::(_decref) Task=`8ebd4e2a-aa9f-45ee-b0ee-655df4c55b10`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:55,405::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:55,406::task::595::TaskManager.Task::(_updateState) Task=`b52bd59a-ddfd-4c49-a813-b764ed9d2b9f`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:55,406::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:55,406::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:55,406::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:55,407::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:55,407::task::1185::TaskManager.Task::(prepare) Task=`b52bd59a-ddfd-4c49-a813-b764ed9d2b9f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:55,407::task::595::TaskManager.Task::(_updateState) Task=`b52bd59a-ddfd-4c49-a813-b764ed9d2b9f`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:55,408::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:55,408::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:55,408::task::990::TaskManager.Task::(_decref) Task=`b52bd59a-ddfd-4c49-a813-b764ed9d2b9f`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:58:56,023::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:58:56,023::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:58:56,023::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:58:56,024::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:58:56,079::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:58:56,080::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:58:56,081::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:58:56,081::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:58:56,293::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:58:56,296::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:58:56,332::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:58:56,333::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:58:56,354::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:58:56,356::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:58:56,356::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:58:56,386::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:58:56,387::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:58:56,414::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:56,415::task::595::TaskManager.Task::(_updateState) Task=`cef5d27f-2dbc-40e1-ba71-e77cd3dbe085`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:56,415::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:56,416::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:56,416::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:56,416::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:56,417::task::1185::TaskManager.Task::(prepare) Task=`cef5d27f-2dbc-40e1-ba71-e77cd3dbe085`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:56,417::task::595::TaskManager.Task::(_updateState) Task=`cef5d27f-2dbc-40e1-ba71-e77cd3dbe085`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:56,417::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:56,418::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:56,418::task::990::TaskManager.Task::(_decref) Task=`cef5d27f-2dbc-40e1-ba71-e77cd3dbe085`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:57,423::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:57,424::task::595::TaskManager.Task::(_updateState) Task=`d56abd21-503a-4ecc-82f4-8240030dcbc2`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:57,424::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:57,425::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:57,425::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:57,425::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:57,426::task::1185::TaskManager.Task::(prepare) Task=`d56abd21-503a-4ecc-82f4-8240030dcbc2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:57,426::task::595::TaskManager.Task::(_updateState) Task=`d56abd21-503a-4ecc-82f4-8240030dcbc2`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:57,426::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:57,427::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:57,427::task::990::TaskManager.Task::(_decref) Task=`d56abd21-503a-4ecc-82f4-8240030dcbc2`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:58,433::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:58,434::task::595::TaskManager.Task::(_updateState) Task=`daff0f4a-b3c4-45a0-8f76-ea5f272297fb`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:58,434::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:58,435::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:58,435::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:58,435::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:58,436::task::1185::TaskManager.Task::(prepare) Task=`daff0f4a-b3c4-45a0-8f76-ea5f272297fb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:58,436::task::595::TaskManager.Task::(_updateState) Task=`daff0f4a-b3c4-45a0-8f76-ea5f272297fb`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:58,436::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:58,437::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:58,437::task::990::TaskManager.Task::(_decref) Task=`daff0f4a-b3c4-45a0-8f76-ea5f272297fb`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:58:59,442::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:58:59,443::task::595::TaskManager.Task::(_updateState) Task=`8f3a36a8-628d-4dad-bf78-b70bf8ade1b3`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:58:59,443::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:58:59,444::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:58:59,444::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:58:59,444::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:59,445::task::1185::TaskManager.Task::(prepare) Task=`8f3a36a8-628d-4dad-bf78-b70bf8ade1b3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:58:59,445::task::595::TaskManager.Task::(_updateState) Task=`8f3a36a8-628d-4dad-bf78-b70bf8ade1b3`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:58:59,445::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:58:59,446::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:58:59,446::task::990::TaskManager.Task::(_decref) Task=`8f3a36a8-628d-4dad-bf78-b70bf8ade1b3`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:00,450::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:00,451::task::595::TaskManager.Task::(_updateState) Task=`3eb90c3d-3702-4b1b-8d5f-5a52c3f2af2e`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:00,452::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:00,452::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:00,452::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:00,453::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:00,453::task::1185::TaskManager.Task::(prepare) Task=`3eb90c3d-3702-4b1b-8d5f-5a52c3f2af2e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:00,453::task::595::TaskManager.Task::(_updateState) Task=`3eb90c3d-3702-4b1b-8d5f-5a52c3f2af2e`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:00,454::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:00,454::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:00,454::task::990::TaskManager.Task::(_decref) Task=`3eb90c3d-3702-4b1b-8d5f-5a52c3f2af2e`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:01,498::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:01,498::task::595::TaskManager.Task::(_updateState) Task=`55443542-eb2d-4d8c-af17-c36aed10282a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:01,499::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:01,499::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:01,499::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:01,500::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:01,500::task::1185::TaskManager.Task::(prepare) Task=`55443542-eb2d-4d8c-af17-c36aed10282a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:01,500::task::595::TaskManager.Task::(_updateState) Task=`55443542-eb2d-4d8c-af17-c36aed10282a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:01,501::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:01,501::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:01,501::task::990::TaskManager.Task::(_decref) Task=`55443542-eb2d-4d8c-af17-c36aed10282a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:02,506::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:02,507::task::595::TaskManager.Task::(_updateState) Task=`e7b077fd-1e5f-4159-863b-9bc66e8aa666`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:02,507::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:02,508::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:02,508::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:02,508::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:02,509::task::1185::TaskManager.Task::(prepare) Task=`e7b077fd-1e5f-4159-863b-9bc66e8aa666`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:02,509::task::595::TaskManager.Task::(_updateState) Task=`e7b077fd-1e5f-4159-863b-9bc66e8aa666`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:02,509::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:02,509::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:02,510::task::990::TaskManager.Task::(_decref) Task=`e7b077fd-1e5f-4159-863b-9bc66e8aa666`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:59:03,141::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:59:03,165::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000312911 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:59:03,276::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:59:03,288::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000422252 s, 1.7 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:59:03,515::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:03,516::task::595::TaskManager.Task::(_updateState) Task=`d02b3a6e-a49d-44da-8601-950ecffc5aae`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:03,516::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:03,516::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:03,517::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:03,517::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:03,517::task::1185::TaskManager.Task::(prepare) Task=`d02b3a6e-a49d-44da-8601-950ecffc5aae`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:03,518::task::595::TaskManager.Task::(_updateState) Task=`d02b3a6e-a49d-44da-8601-950ecffc5aae`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:03,518::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:03,518::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:03,519::task::990::TaskManager.Task::(_decref) Task=`d02b3a6e-a49d-44da-8601-950ecffc5aae`::ref 0 aborting False >Thread-29::DEBUG::2014-04-11 22:59:03,618::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:59:03,652::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000466251 s, 1.6 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:59:03,679::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:59:03,692::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00034126 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:59:04,524::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:04,525::task::595::TaskManager.Task::(_updateState) Task=`465ac12e-8e89-439d-8c4c-5e55aa2c9875`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:04,526::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:04,526::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:04,527::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:04,527::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:04,527::task::1185::TaskManager.Task::(prepare) Task=`465ac12e-8e89-439d-8c4c-5e55aa2c9875`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:04,528::task::595::TaskManager.Task::(_updateState) Task=`465ac12e-8e89-439d-8c4c-5e55aa2c9875`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:04,528::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:04,528::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:04,529::task::990::TaskManager.Task::(_decref) Task=`465ac12e-8e89-439d-8c4c-5e55aa2c9875`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:05,534::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:05,535::task::595::TaskManager.Task::(_updateState) Task=`ca104189-6abd-41ad-8613-57ea0a6e97e7`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:05,535::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:05,535::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:05,536::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:05,536::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:05,536::task::1185::TaskManager.Task::(prepare) Task=`ca104189-6abd-41ad-8613-57ea0a6e97e7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:05,537::task::595::TaskManager.Task::(_updateState) Task=`ca104189-6abd-41ad-8613-57ea0a6e97e7`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:05,537::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:05,537::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:05,538::task::990::TaskManager.Task::(_decref) Task=`ca104189-6abd-41ad-8613-57ea0a6e97e7`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:59:06,334::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:59:06,334::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:59:06,347::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:59:06,348::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:59:06,388::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:59:06,389::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:59:06,389::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:59:06,390::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 22:59:06,513::task::595::TaskManager.Task::(_updateState) Task=`8148c05e-7eef-4e48-affc-40f8413e6ab1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:06,514::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:59:06,517::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000422252', 'lastCheck': '3.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000312911', 'lastCheck': '3.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000466251', 'lastCheck': '2.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '95.4', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00034126', 'lastCheck': '2.8', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:59:06,517::task::1185::TaskManager.Task::(prepare) Task=`8148c05e-7eef-4e48-affc-40f8413e6ab1`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000422252', 'lastCheck': '3.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000312911', 'lastCheck': '3.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000466251', 'lastCheck': '2.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '95.4', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00034126', 'lastCheck': '2.8', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:59:06,521::task::595::TaskManager.Task::(_updateState) Task=`8148c05e-7eef-4e48-affc-40f8413e6ab1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:06,525::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:06,527::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:06,528::task::990::TaskManager.Task::(_decref) Task=`8148c05e-7eef-4e48-affc-40f8413e6ab1`::ref 0 aborting False >Thread-15::DEBUG::2014-04-11 22:59:06,543::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-15::DEBUG::2014-04-11 22:59:06,544::task::595::TaskManager.Task::(_updateState) Task=`f664a9a1-c956-4f67-aaaf-aa4280570da5`::moving from state init -> state preparing >Thread-15::INFO::2014-04-11 22:59:06,545::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-15::DEBUG::2014-04-11 22:59:06,565::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-15::DEBUG::2014-04-11 22:59:06,586::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-15::INFO::2014-04-11 22:59:06,593::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-15::DEBUG::2014-04-11 22:59:06,593::task::1185::TaskManager.Task::(prepare) Task=`f664a9a1-c956-4f67-aaaf-aa4280570da5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-15::DEBUG::2014-04-11 22:59:06,594::task::595::TaskManager.Task::(_updateState) Task=`f664a9a1-c956-4f67-aaaf-aa4280570da5`::moving from state preparing -> state finished >Thread-15::DEBUG::2014-04-11 22:59:06,594::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15::DEBUG::2014-04-11 22:59:06,595::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15::DEBUG::2014-04-11 22:59:06,596::task::990::TaskManager.Task::(_decref) Task=`f664a9a1-c956-4f67-aaaf-aa4280570da5`::ref 0 aborting False >Thread-39::DEBUG::2014-04-11 22:59:06,641::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:59:06,643::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:59:06,685::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:59:06,688::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:59:06,689::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:59:06,693::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:59:06,693::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:59:06,722::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:59:06,723::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:59:07,603::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:07,603::task::595::TaskManager.Task::(_updateState) Task=`6a4e9f54-fb9e-4d66-8caf-8db62df964ce`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:07,604::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:07,604::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:07,605::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:07,605::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:07,605::task::1185::TaskManager.Task::(prepare) Task=`6a4e9f54-fb9e-4d66-8caf-8db62df964ce`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:07,606::task::595::TaskManager.Task::(_updateState) Task=`6a4e9f54-fb9e-4d66-8caf-8db62df964ce`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:07,606::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:07,606::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:07,607::task::990::TaskManager.Task::(_decref) Task=`6a4e9f54-fb9e-4d66-8caf-8db62df964ce`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:08,615::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:08,619::task::595::TaskManager.Task::(_updateState) Task=`26c05cfc-757a-4539-a5d2-017b512322d7`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:08,622::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:08,623::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:08,625::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:08,626::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:08,628::task::1185::TaskManager.Task::(prepare) Task=`26c05cfc-757a-4539-a5d2-017b512322d7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:08,630::task::595::TaskManager.Task::(_updateState) Task=`26c05cfc-757a-4539-a5d2-017b512322d7`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:08,631::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:08,631::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:08,634::task::990::TaskManager.Task::(_decref) Task=`26c05cfc-757a-4539-a5d2-017b512322d7`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:09,639::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:09,640::task::595::TaskManager.Task::(_updateState) Task=`da7ff334-b63c-4b9a-aa42-c35d3662bee3`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:09,640::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:09,640::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:09,640::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:09,641::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:09,641::task::1185::TaskManager.Task::(prepare) Task=`da7ff334-b63c-4b9a-aa42-c35d3662bee3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:09,641::task::595::TaskManager.Task::(_updateState) Task=`da7ff334-b63c-4b9a-aa42-c35d3662bee3`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:09,642::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:09,642::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:09,642::task::990::TaskManager.Task::(_decref) Task=`da7ff334-b63c-4b9a-aa42-c35d3662bee3`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:10,647::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:10,647::task::595::TaskManager.Task::(_updateState) Task=`f75bbfa4-db24-41c4-a9af-745793adb20c`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:10,648::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:10,648::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:10,648::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:10,648::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:10,649::task::1185::TaskManager.Task::(prepare) Task=`f75bbfa4-db24-41c4-a9af-745793adb20c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:10,649::task::595::TaskManager.Task::(_updateState) Task=`f75bbfa4-db24-41c4-a9af-745793adb20c`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:10,649::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:10,650::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:10,650::task::990::TaskManager.Task::(_decref) Task=`f75bbfa4-db24-41c4-a9af-745793adb20c`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:11,654::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:11,693::task::595::TaskManager.Task::(_updateState) Task=`9096f25e-d876-4467-bb09-3e1753b75059`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:11,694::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:11,694::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:11,694::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:11,695::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:11,695::task::1185::TaskManager.Task::(prepare) Task=`9096f25e-d876-4467-bb09-3e1753b75059`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:11,695::task::595::TaskManager.Task::(_updateState) Task=`9096f25e-d876-4467-bb09-3e1753b75059`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:11,696::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:11,696::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:11,696::task::990::TaskManager.Task::(_decref) Task=`9096f25e-d876-4467-bb09-3e1753b75059`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:12,701::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:12,701::task::595::TaskManager.Task::(_updateState) Task=`e2df0a0e-ff8e-42aa-9812-cdb30b19fe62`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:12,701::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:12,702::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:12,702::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:12,702::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:12,703::task::1185::TaskManager.Task::(prepare) Task=`e2df0a0e-ff8e-42aa-9812-cdb30b19fe62`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:12,703::task::595::TaskManager.Task::(_updateState) Task=`e2df0a0e-ff8e-42aa-9812-cdb30b19fe62`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:12,703::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:12,703::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:12,704::task::990::TaskManager.Task::(_decref) Task=`e2df0a0e-ff8e-42aa-9812-cdb30b19fe62`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:59:13,190::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:59:13,203::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000413842 s, 892 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:59:13,323::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:59:13,335::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.00048539 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:59:13,686::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-14::DEBUG::2014-04-11 22:59:13,715::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:13,722::task::595::TaskManager.Task::(_updateState) Task=`9bd60b08-8f44-44bb-a4e6-2aa554879e49`::moving from state init -> state preparing >Thread-29::DEBUG::2014-04-11 22:59:13,726::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000552306 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14::INFO::2014-04-11 22:59:13,727::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:13,732::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:13,736::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:13,740::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:13,745::task::1185::TaskManager.Task::(prepare) Task=`9bd60b08-8f44-44bb-a4e6-2aa554879e49`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:13,747::task::595::TaskManager.Task::(_updateState) Task=`9bd60b08-8f44-44bb-a4e6-2aa554879e49`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:13,750::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:13,752::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:13,755::task::990::TaskManager.Task::(_decref) Task=`9bd60b08-8f44-44bb-a4e6-2aa554879e49`::ref 0 aborting False >Thread-46::DEBUG::2014-04-11 22:59:13,782::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:59:13,794::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000348219 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:59:14,767::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:14,768::task::595::TaskManager.Task::(_updateState) Task=`14e1e225-744d-41b8-a1d7-d6a25aa4a4d4`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:14,768::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:14,769::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:14,769::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:14,770::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:14,770::task::1185::TaskManager.Task::(prepare) Task=`14e1e225-744d-41b8-a1d7-d6a25aa4a4d4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:14,770::task::595::TaskManager.Task::(_updateState) Task=`14e1e225-744d-41b8-a1d7-d6a25aa4a4d4`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:14,771::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:14,771::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:14,771::task::990::TaskManager.Task::(_decref) Task=`14e1e225-744d-41b8-a1d7-d6a25aa4a4d4`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:15,776::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:15,777::task::595::TaskManager.Task::(_updateState) Task=`279cbc80-40bc-44e3-b4af-ff4b39e48aef`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:15,777::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:15,777::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:15,777::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:15,778::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:15,778::task::1185::TaskManager.Task::(prepare) Task=`279cbc80-40bc-44e3-b4af-ff4b39e48aef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:15,778::task::595::TaskManager.Task::(_updateState) Task=`279cbc80-40bc-44e3-b4af-ff4b39e48aef`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:15,779::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:15,779::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:15,779::task::990::TaskManager.Task::(_decref) Task=`279cbc80-40bc-44e3-b4af-ff4b39e48aef`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:59:16,691::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:59:16,692::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:59:16,710::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:59:16,711::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:59:16,724::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:59:16,725::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:59:16,725::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:59:16,726::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 22:59:16,786::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:16,788::task::595::TaskManager.Task::(_updateState) Task=`c51d021a-d6c1-452a-a715-1e5980a5854b`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:16,789::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:16,789::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:16,790::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:16,790::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:16,791::task::1185::TaskManager.Task::(prepare) Task=`c51d021a-d6c1-452a-a715-1e5980a5854b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:16,791::task::595::TaskManager.Task::(_updateState) Task=`c51d021a-d6c1-452a-a715-1e5980a5854b`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:16,792::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:16,792::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:16,792::task::990::TaskManager.Task::(_decref) Task=`c51d021a-d6c1-452a-a715-1e5980a5854b`::ref 0 aborting False >Thread-39::DEBUG::2014-04-11 22:59:16,983::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:59:16,984::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-33::DEBUG::2014-04-11 22:59:17,025::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-39::ERROR::2014-04-11 22:59:17,029::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:59:17,033::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::WARNING::2014-04-11 22:59:17,037::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:59:17,038::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:59:17,069::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:59:17,069::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:59:17,798::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:17,799::task::595::TaskManager.Task::(_updateState) Task=`c437fb59-7997-4c31-8669-72eee0cb4d59`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:17,799::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:17,800::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:17,800::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:17,800::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:17,801::task::1185::TaskManager.Task::(prepare) Task=`c437fb59-7997-4c31-8669-72eee0cb4d59`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:17,801::task::595::TaskManager.Task::(_updateState) Task=`c437fb59-7997-4c31-8669-72eee0cb4d59`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:17,801::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:17,802::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:17,802::task::990::TaskManager.Task::(_decref) Task=`c437fb59-7997-4c31-8669-72eee0cb4d59`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:18,808::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:18,809::task::595::TaskManager.Task::(_updateState) Task=`f57d51d3-47f7-4771-b6b9-03deacb0a4d0`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:18,810::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:18,810::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:18,810::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:18,811::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:18,811::task::1185::TaskManager.Task::(prepare) Task=`f57d51d3-47f7-4771-b6b9-03deacb0a4d0`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:18,811::task::595::TaskManager.Task::(_updateState) Task=`f57d51d3-47f7-4771-b6b9-03deacb0a4d0`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:18,811::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:18,812::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:18,812::task::990::TaskManager.Task::(_decref) Task=`f57d51d3-47f7-4771-b6b9-03deacb0a4d0`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:19,817::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:19,817::task::595::TaskManager.Task::(_updateState) Task=`8a3f0133-36a9-41b4-8b5e-d4a3d2de03e7`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:19,818::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:19,818::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:19,818::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:19,819::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:19,819::task::1185::TaskManager.Task::(prepare) Task=`8a3f0133-36a9-41b4-8b5e-d4a3d2de03e7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:19,819::task::595::TaskManager.Task::(_updateState) Task=`8a3f0133-36a9-41b4-8b5e-d4a3d2de03e7`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:19,820::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:19,820::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:19,820::task::990::TaskManager.Task::(_decref) Task=`8a3f0133-36a9-41b4-8b5e-d4a3d2de03e7`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:20,825::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:20,825::task::595::TaskManager.Task::(_updateState) Task=`67f44e0d-6f62-4b4a-a1a6-403e89c5c489`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:20,826::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:20,826::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:20,826::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:20,827::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:20,827::task::1185::TaskManager.Task::(prepare) Task=`67f44e0d-6f62-4b4a-a1a6-403e89c5c489`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:20,827::task::595::TaskManager.Task::(_updateState) Task=`67f44e0d-6f62-4b4a-a1a6-403e89c5c489`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:20,827::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:20,828::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:20,828::task::990::TaskManager.Task::(_decref) Task=`67f44e0d-6f62-4b4a-a1a6-403e89c5c489`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:21,833::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:21,846::task::595::TaskManager.Task::(_updateState) Task=`d720b8b8-4ba5-4f13-a0de-2e7768c3909b`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:21,846::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:21,847::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:21,847::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:21,847::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:21,847::task::1185::TaskManager.Task::(prepare) Task=`d720b8b8-4ba5-4f13-a0de-2e7768c3909b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:21,848::task::595::TaskManager.Task::(_updateState) Task=`d720b8b8-4ba5-4f13-a0de-2e7768c3909b`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:21,848::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:21,848::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:21,849::task::990::TaskManager.Task::(_decref) Task=`d720b8b8-4ba5-4f13-a0de-2e7768c3909b`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:21,928::task::595::TaskManager.Task::(_updateState) Task=`4d955a1b-6071-4000-aff4-486c9d97de17`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:21,929::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:59:21,929::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00048539', 'lastCheck': '8.6', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413842', 'lastCheck': '8.7', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.9', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000552306', 'lastCheck': '8.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '110.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000348219', 'lastCheck': '8.1', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:59:21,930::task::1185::TaskManager.Task::(prepare) Task=`4d955a1b-6071-4000-aff4-486c9d97de17`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00048539', 'lastCheck': '8.6', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413842', 'lastCheck': '8.7', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.9', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000552306', 'lastCheck': '8.2', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '110.8', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000348219', 'lastCheck': '8.1', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:59:21,930::task::595::TaskManager.Task::(_updateState) Task=`4d955a1b-6071-4000-aff4-486c9d97de17`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:21,930::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:21,931::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:21,931::task::990::TaskManager.Task::(_decref) Task=`4d955a1b-6071-4000-aff4-486c9d97de17`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:22,854::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:22,854::task::595::TaskManager.Task::(_updateState) Task=`12a63371-6eae-4512-881b-368421051cc6`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:22,855::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:22,855::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:22,855::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:22,856::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:22,856::task::1185::TaskManager.Task::(prepare) Task=`12a63371-6eae-4512-881b-368421051cc6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:22,856::task::595::TaskManager.Task::(_updateState) Task=`12a63371-6eae-4512-881b-368421051cc6`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:22,856::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:22,857::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:22,857::task::990::TaskManager.Task::(_decref) Task=`12a63371-6eae-4512-881b-368421051cc6`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:59:23,226::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:59:23,239::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000335394 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:59:23,372::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:59:23,384::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000413523 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:59:23,769::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:59:23,783::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000492008 s, 1.5 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:59:23,824::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:59:23,836::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000364182 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:59:23,863::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:23,864::task::595::TaskManager.Task::(_updateState) Task=`64412dc9-88a6-439d-82c2-70c07e0bd7eb`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:23,864::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:23,864::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:23,865::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:23,865::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:23,865::task::1185::TaskManager.Task::(prepare) Task=`64412dc9-88a6-439d-82c2-70c07e0bd7eb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:23,866::task::595::TaskManager.Task::(_updateState) Task=`64412dc9-88a6-439d-82c2-70c07e0bd7eb`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:23,866::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:23,867::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:23,867::task::990::TaskManager.Task::(_decref) Task=`64412dc9-88a6-439d-82c2-70c07e0bd7eb`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:24,872::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:24,873::task::595::TaskManager.Task::(_updateState) Task=`56697163-1bab-41e3-b292-a435847c8317`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:24,873::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:24,873::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:24,874::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:24,874::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:24,874::task::1185::TaskManager.Task::(prepare) Task=`56697163-1bab-41e3-b292-a435847c8317`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:24,875::task::595::TaskManager.Task::(_updateState) Task=`56697163-1bab-41e3-b292-a435847c8317`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:24,875::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:24,875::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:24,875::task::990::TaskManager.Task::(_decref) Task=`56697163-1bab-41e3-b292-a435847c8317`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:25,880::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:25,881::task::595::TaskManager.Task::(_updateState) Task=`c0b33738-54be-4791-af29-5b952bc88171`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:25,881::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:25,881::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:25,882::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:25,882::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:25,882::task::1185::TaskManager.Task::(prepare) Task=`c0b33738-54be-4791-af29-5b952bc88171`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:25,883::task::595::TaskManager.Task::(_updateState) Task=`c0b33738-54be-4791-af29-5b952bc88171`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:25,883::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:25,883::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:25,884::task::990::TaskManager.Task::(_decref) Task=`c0b33738-54be-4791-af29-5b952bc88171`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:26,888::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:26,891::task::595::TaskManager.Task::(_updateState) Task=`291748f2-e871-484c-a1d2-e2bb7d8faf7c`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:26,892::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:26,892::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:26,892::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:26,892::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:26,893::task::1185::TaskManager.Task::(prepare) Task=`291748f2-e871-484c-a1d2-e2bb7d8faf7c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:26,893::task::595::TaskManager.Task::(_updateState) Task=`291748f2-e871-484c-a1d2-e2bb7d8faf7c`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:26,893::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:26,893::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:26,894::task::990::TaskManager.Task::(_decref) Task=`291748f2-e871-484c-a1d2-e2bb7d8faf7c`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:59:27,038::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:59:27,038::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:59:27,039::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:59:27,039::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:59:27,071::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:59:27,071::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:59:27,072::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:59:27,073::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:59:27,308::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:59:27,311::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:59:27,344::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:59:27,347::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:59:27,353::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:59:27,355::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:59:27,355::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:59:27,384::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:59:27,385::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:59:27,899::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:27,900::task::595::TaskManager.Task::(_updateState) Task=`04ce6a26-8f41-4003-96dc-d4a7528ef197`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:27,901::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:27,901::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:27,901::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:27,901::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:27,902::task::1185::TaskManager.Task::(prepare) Task=`04ce6a26-8f41-4003-96dc-d4a7528ef197`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:27,902::task::595::TaskManager.Task::(_updateState) Task=`04ce6a26-8f41-4003-96dc-d4a7528ef197`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:27,902::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:27,903::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:27,903::task::990::TaskManager.Task::(_decref) Task=`04ce6a26-8f41-4003-96dc-d4a7528ef197`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:28,909::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:28,910::task::595::TaskManager.Task::(_updateState) Task=`995728e3-469d-474e-910c-a1bd25851cdc`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:28,911::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:28,911::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:28,911::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:28,912::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:28,912::task::1185::TaskManager.Task::(prepare) Task=`995728e3-469d-474e-910c-a1bd25851cdc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:28,912::task::595::TaskManager.Task::(_updateState) Task=`995728e3-469d-474e-910c-a1bd25851cdc`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:28,913::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:28,913::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:28,913::task::990::TaskManager.Task::(_decref) Task=`995728e3-469d-474e-910c-a1bd25851cdc`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:29,918::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:29,919::task::595::TaskManager.Task::(_updateState) Task=`62e523e6-f376-44de-91ef-46cabfa2baeb`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:29,920::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:29,920::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:29,920::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:29,920::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:29,921::task::1185::TaskManager.Task::(prepare) Task=`62e523e6-f376-44de-91ef-46cabfa2baeb`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:29,921::task::595::TaskManager.Task::(_updateState) Task=`62e523e6-f376-44de-91ef-46cabfa2baeb`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:29,921::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:29,922::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:29,922::task::990::TaskManager.Task::(_decref) Task=`62e523e6-f376-44de-91ef-46cabfa2baeb`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:30,927::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:30,927::task::595::TaskManager.Task::(_updateState) Task=`8687d34c-140c-441a-9b55-2ac74176d704`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:30,928::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:30,928::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:30,928::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:30,929::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:30,929::task::1185::TaskManager.Task::(prepare) Task=`8687d34c-140c-441a-9b55-2ac74176d704`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:30,929::task::595::TaskManager.Task::(_updateState) Task=`8687d34c-140c-441a-9b55-2ac74176d704`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:30,929::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:30,930::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:30,930::task::990::TaskManager.Task::(_decref) Task=`8687d34c-140c-441a-9b55-2ac74176d704`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:31,935::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:31,935::task::595::TaskManager.Task::(_updateState) Task=`b2ea4ad3-ff81-4917-9a6c-6b6b67cc0930`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:31,935::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:31,936::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:31,936::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:31,936::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:31,937::task::1185::TaskManager.Task::(prepare) Task=`b2ea4ad3-ff81-4917-9a6c-6b6b67cc0930`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:31,937::task::595::TaskManager.Task::(_updateState) Task=`b2ea4ad3-ff81-4917-9a6c-6b6b67cc0930`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:31,937::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:31,937::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:31,938::task::990::TaskManager.Task::(_decref) Task=`b2ea4ad3-ff81-4917-9a6c-6b6b67cc0930`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:32,942::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:32,943::task::595::TaskManager.Task::(_updateState) Task=`b2f8b1aa-a2ce-42ed-a622-1222b94d82ca`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:32,943::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:32,944::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:32,944::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:32,944::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:32,944::task::1185::TaskManager.Task::(prepare) Task=`b2f8b1aa-a2ce-42ed-a622-1222b94d82ca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:32,945::task::595::TaskManager.Task::(_updateState) Task=`b2f8b1aa-a2ce-42ed-a622-1222b94d82ca`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:32,945::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:32,945::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:32,946::task::990::TaskManager.Task::(_decref) Task=`b2f8b1aa-a2ce-42ed-a622-1222b94d82ca`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:59:33,264::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:59:33,277::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000405978 s, 909 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:59:33,414::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:59:33,427::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000499164 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:59:33,809::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:59:33,822::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000487717 s, 1.5 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:59:33,866::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:59:33,878::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000450471 s, 1.7 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:59:33,951::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:33,952::task::595::TaskManager.Task::(_updateState) Task=`748051c5-d97d-442b-810c-d93447a592b7`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:33,953::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:33,953::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:33,953::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:33,954::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:33,954::task::1185::TaskManager.Task::(prepare) Task=`748051c5-d97d-442b-810c-d93447a592b7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:33,954::task::595::TaskManager.Task::(_updateState) Task=`748051c5-d97d-442b-810c-d93447a592b7`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:33,955::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:33,955::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:33,955::task::990::TaskManager.Task::(_decref) Task=`748051c5-d97d-442b-810c-d93447a592b7`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:34,961::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:34,962::task::595::TaskManager.Task::(_updateState) Task=`0331cdf6-edf9-4a0a-a4b1-679b0b7483fc`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:34,963::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:34,963::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:34,963::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:34,964::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:34,964::task::1185::TaskManager.Task::(prepare) Task=`0331cdf6-edf9-4a0a-a4b1-679b0b7483fc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:34,964::task::595::TaskManager.Task::(_updateState) Task=`0331cdf6-edf9-4a0a-a4b1-679b0b7483fc`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:34,965::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:34,965::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:34,965::task::990::TaskManager.Task::(_decref) Task=`0331cdf6-edf9-4a0a-a4b1-679b0b7483fc`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:35,971::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:35,971::task::595::TaskManager.Task::(_updateState) Task=`cd76867b-af37-4672-bcb9-9e9788d3a955`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:35,972::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:35,972::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:35,972::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:35,973::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:35,973::task::1185::TaskManager.Task::(prepare) Task=`cd76867b-af37-4672-bcb9-9e9788d3a955`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:35,974::task::595::TaskManager.Task::(_updateState) Task=`cd76867b-af37-4672-bcb9-9e9788d3a955`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:35,974::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:35,974::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:35,975::task::990::TaskManager.Task::(_decref) Task=`cd76867b-af37-4672-bcb9-9e9788d3a955`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:36,980::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:36,980::task::595::TaskManager.Task::(_updateState) Task=`73018eec-c27b-4d31-b964-a1bad22dae80`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:36,981::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:36,981::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:36,981::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:36,981::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:36,982::task::1185::TaskManager.Task::(prepare) Task=`73018eec-c27b-4d31-b964-a1bad22dae80`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:36,982::task::595::TaskManager.Task::(_updateState) Task=`73018eec-c27b-4d31-b964-a1bad22dae80`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:36,982::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:36,983::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:36,983::task::990::TaskManager.Task::(_decref) Task=`73018eec-c27b-4d31-b964-a1bad22dae80`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:37,308::task::595::TaskManager.Task::(_updateState) Task=`1952d4a0-f9ac-4aa7-9239-400326d6856d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:37,308::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:59:37,309::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000499164', 'lastCheck': '3.9', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000405978', 'lastCheck': '4.0', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.9', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000487717', 'lastCheck': '3.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.0', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '126.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000450471', 'lastCheck': '3.4', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:59:37,310::task::1185::TaskManager.Task::(prepare) Task=`1952d4a0-f9ac-4aa7-9239-400326d6856d`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000499164', 'lastCheck': '3.9', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000405978', 'lastCheck': '4.0', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.9', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000487717', 'lastCheck': '3.5', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.0', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '126.2', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000450471', 'lastCheck': '3.4', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:59:37,310::task::595::TaskManager.Task::(_updateState) Task=`1952d4a0-f9ac-4aa7-9239-400326d6856d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:37,310::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:37,311::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:37,311::task::990::TaskManager.Task::(_decref) Task=`1952d4a0-f9ac-4aa7-9239-400326d6856d`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:59:37,353::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:59:37,353::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:59:37,354::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:59:37,354::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:59:37,386::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:59:37,387::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:59:37,387::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:59:37,388::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:59:37,623::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:59:37,626::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::ERROR::2014-04-11 22:59:37,661::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:59:37,661::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-33::DEBUG::2014-04-11 22:59:37,667::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:59:37,669::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::DEBUG::2014-04-11 22:59:37,670::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:59:37,701::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:59:37,701::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 22:59:37,989::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:38,011::task::595::TaskManager.Task::(_updateState) Task=`4391fc55-397d-4251-9544-df4b0502acf1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:38,012::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:38,012::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:38,012::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:38,013::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:38,013::task::1185::TaskManager.Task::(prepare) Task=`4391fc55-397d-4251-9544-df4b0502acf1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:38,013::task::595::TaskManager.Task::(_updateState) Task=`4391fc55-397d-4251-9544-df4b0502acf1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:38,014::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:38,014::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:38,014::task::990::TaskManager.Task::(_decref) Task=`4391fc55-397d-4251-9544-df4b0502acf1`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:39,031::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:39,032::task::595::TaskManager.Task::(_updateState) Task=`1142807a-1b4b-4159-9a0a-87a0b29a5b95`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:39,032::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:39,033::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:39,033::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:39,033::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:39,034::task::1185::TaskManager.Task::(prepare) Task=`1142807a-1b4b-4159-9a0a-87a0b29a5b95`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:39,034::task::595::TaskManager.Task::(_updateState) Task=`1142807a-1b4b-4159-9a0a-87a0b29a5b95`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:39,034::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:39,035::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:39,035::task::990::TaskManager.Task::(_decref) Task=`1142807a-1b4b-4159-9a0a-87a0b29a5b95`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:40,041::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:40,041::task::595::TaskManager.Task::(_updateState) Task=`b9acd7d8-aef1-4296-8952-a75fefe8ab70`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:40,042::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:40,042::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:40,042::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:40,042::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:40,043::task::1185::TaskManager.Task::(prepare) Task=`b9acd7d8-aef1-4296-8952-a75fefe8ab70`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:40,043::task::595::TaskManager.Task::(_updateState) Task=`b9acd7d8-aef1-4296-8952-a75fefe8ab70`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:40,043::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:40,044::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:40,044::task::990::TaskManager.Task::(_decref) Task=`b9acd7d8-aef1-4296-8952-a75fefe8ab70`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:41,048::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:41,049::task::595::TaskManager.Task::(_updateState) Task=`16f2f6ed-6f15-4069-96f4-b6c480941d10`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:41,049::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:41,050::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:41,050::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:41,050::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:41,051::task::1185::TaskManager.Task::(prepare) Task=`16f2f6ed-6f15-4069-96f4-b6c480941d10`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:41,051::task::595::TaskManager.Task::(_updateState) Task=`16f2f6ed-6f15-4069-96f4-b6c480941d10`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:41,051::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:41,051::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:41,052::task::990::TaskManager.Task::(_decref) Task=`16f2f6ed-6f15-4069-96f4-b6c480941d10`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:42,056::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:42,057::task::595::TaskManager.Task::(_updateState) Task=`61c14be9-3859-411a-9ce0-53f84cc9c669`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:42,057::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:42,058::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:42,058::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:42,058::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:42,059::task::1185::TaskManager.Task::(prepare) Task=`61c14be9-3859-411a-9ce0-53f84cc9c669`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:42,059::task::595::TaskManager.Task::(_updateState) Task=`61c14be9-3859-411a-9ce0-53f84cc9c669`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:42,059::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:42,059::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:42,060::task::990::TaskManager.Task::(_decref) Task=`61c14be9-3859-411a-9ce0-53f84cc9c669`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:43,064::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:43,081::task::595::TaskManager.Task::(_updateState) Task=`6253d6b8-7335-46a6-8acb-cb3bdec81ee6`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:43,082::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:43,082::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:43,082::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:43,082::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:43,083::task::1185::TaskManager.Task::(prepare) Task=`6253d6b8-7335-46a6-8acb-cb3bdec81ee6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:43,083::task::595::TaskManager.Task::(_updateState) Task=`6253d6b8-7335-46a6-8acb-cb3bdec81ee6`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:43,083::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:43,084::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:43,084::task::990::TaskManager.Task::(_decref) Task=`6253d6b8-7335-46a6-8acb-cb3bdec81ee6`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:59:43,302::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:59:43,315::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000346167 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:59:43,456::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:59:43,469::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000485038 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:59:43,851::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:59:43,864::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00051056 s, 1.5 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:59:43,908::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:59:43,921::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000379966 s, 2.0 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:59:44,089::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:44,090::task::595::TaskManager.Task::(_updateState) Task=`6dc1a420-0dbf-4fe4-a1d3-bc68e2c4ee3a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:44,091::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:44,091::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:44,091::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:44,092::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:44,092::task::1185::TaskManager.Task::(prepare) Task=`6dc1a420-0dbf-4fe4-a1d3-bc68e2c4ee3a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:44,092::task::595::TaskManager.Task::(_updateState) Task=`6dc1a420-0dbf-4fe4-a1d3-bc68e2c4ee3a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:44,093::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:44,093::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:44,093::task::990::TaskManager.Task::(_decref) Task=`6dc1a420-0dbf-4fe4-a1d3-bc68e2c4ee3a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:45,099::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:45,100::task::595::TaskManager.Task::(_updateState) Task=`00a6dabd-201b-4c1a-83b1-37e248cbec82`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:45,100::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:45,100::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:45,101::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:45,101::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:45,101::task::1185::TaskManager.Task::(prepare) Task=`00a6dabd-201b-4c1a-83b1-37e248cbec82`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:45,102::task::595::TaskManager.Task::(_updateState) Task=`00a6dabd-201b-4c1a-83b1-37e248cbec82`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:45,102::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:45,102::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:45,103::task::990::TaskManager.Task::(_decref) Task=`00a6dabd-201b-4c1a-83b1-37e248cbec82`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:46,108::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:46,108::task::595::TaskManager.Task::(_updateState) Task=`d2fd0140-4ac3-419f-9d33-8b03e0bee4ca`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:46,109::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:46,109::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:46,109::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:46,109::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:46,110::task::1185::TaskManager.Task::(prepare) Task=`d2fd0140-4ac3-419f-9d33-8b03e0bee4ca`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:46,110::task::595::TaskManager.Task::(_updateState) Task=`d2fd0140-4ac3-419f-9d33-8b03e0bee4ca`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:46,110::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:46,110::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:46,111::task::990::TaskManager.Task::(_decref) Task=`d2fd0140-4ac3-419f-9d33-8b03e0bee4ca`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:47,115::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:47,116::task::595::TaskManager.Task::(_updateState) Task=`3d0ccc1a-5ead-45a1-b0cb-0790c3a02d93`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:47,116::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:47,117::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:47,117::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:47,117::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:47,118::task::1185::TaskManager.Task::(prepare) Task=`3d0ccc1a-5ead-45a1-b0cb-0790c3a02d93`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:47,118::task::595::TaskManager.Task::(_updateState) Task=`3d0ccc1a-5ead-45a1-b0cb-0790c3a02d93`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:47,118::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:47,118::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:47,119::task::990::TaskManager.Task::(_decref) Task=`3d0ccc1a-5ead-45a1-b0cb-0790c3a02d93`::ref 0 aborting False >Thread-39::ERROR::2014-04-11 22:59:47,662::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 22:59:47,663::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 22:59:47,663::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 22:59:47,664::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::ERROR::2014-04-11 22:59:47,703::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:59:47,703::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 22:59:47,704::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-33::DEBUG::2014-04-11 22:59:47,705::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:59:47,986::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-33::DEBUG::2014-04-11 22:59:47,994::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:59:47,997::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-39::WARNING::2014-04-11 22:59:47,996::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::DEBUG::2014-04-11 22:59:48,002::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:59:48,070::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:59:48,079::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-39::ERROR::2014-04-11 22:59:48,082::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:59:48,085::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-14::DEBUG::2014-04-11 22:59:48,124::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:48,125::task::595::TaskManager.Task::(_updateState) Task=`4dda0834-cc2f-447d-b93d-09ce58261a44`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:48,125::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:48,126::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:48,126::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:48,127::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:48,127::task::1185::TaskManager.Task::(prepare) Task=`4dda0834-cc2f-447d-b93d-09ce58261a44`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:48,127::task::595::TaskManager.Task::(_updateState) Task=`4dda0834-cc2f-447d-b93d-09ce58261a44`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:48,128::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:48,128::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:48,128::task::990::TaskManager.Task::(_decref) Task=`4dda0834-cc2f-447d-b93d-09ce58261a44`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:49,134::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:49,136::task::595::TaskManager.Task::(_updateState) Task=`9d9d6015-8d60-42f7-b801-7c17f8471407`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:49,137::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:49,137::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:49,137::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:49,138::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:49,138::task::1185::TaskManager.Task::(prepare) Task=`9d9d6015-8d60-42f7-b801-7c17f8471407`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:49,139::task::595::TaskManager.Task::(_updateState) Task=`9d9d6015-8d60-42f7-b801-7c17f8471407`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:49,139::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:49,139::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:49,140::task::990::TaskManager.Task::(_decref) Task=`9d9d6015-8d60-42f7-b801-7c17f8471407`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:50,145::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:50,145::task::595::TaskManager.Task::(_updateState) Task=`8545d396-ac07-4ab2-b049-062fda093a90`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:50,146::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:50,146::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:50,147::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:50,147::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:50,147::task::1185::TaskManager.Task::(prepare) Task=`8545d396-ac07-4ab2-b049-062fda093a90`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:50,148::task::595::TaskManager.Task::(_updateState) Task=`8545d396-ac07-4ab2-b049-062fda093a90`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:50,148::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:50,148::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:50,149::task::990::TaskManager.Task::(_decref) Task=`8545d396-ac07-4ab2-b049-062fda093a90`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:51,153::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:51,154::task::595::TaskManager.Task::(_updateState) Task=`59c8f34c-12f9-461c-97be-90e9f384c77b`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:51,154::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:51,155::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:51,155::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:51,155::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:51,156::task::1185::TaskManager.Task::(prepare) Task=`59c8f34c-12f9-461c-97be-90e9f384c77b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:51,156::task::595::TaskManager.Task::(_updateState) Task=`59c8f34c-12f9-461c-97be-90e9f384c77b`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:51,156::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:51,157::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:51,157::task::990::TaskManager.Task::(_decref) Task=`59c8f34c-12f9-461c-97be-90e9f384c77b`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:52,162::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:52,162::task::595::TaskManager.Task::(_updateState) Task=`6c5ec55e-4048-4016-a800-235ac621b30f`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:52,163::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:52,163::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:52,163::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:52,164::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:52,164::task::1185::TaskManager.Task::(prepare) Task=`6c5ec55e-4048-4016-a800-235ac621b30f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:52,164::task::595::TaskManager.Task::(_updateState) Task=`6c5ec55e-4048-4016-a800-235ac621b30f`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:52,164::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:52,165::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:52,165::task::990::TaskManager.Task::(_decref) Task=`6c5ec55e-4048-4016-a800-235ac621b30f`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:52,666::task::595::TaskManager.Task::(_updateState) Task=`9e19003d-90b9-4a2a-9ed9-da678d55463d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:52,667::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 22:59:52,667::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000485038', 'lastCheck': '9.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346167', 'lastCheck': '9.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00051056', 'lastCheck': '8.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.6', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '141.6', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000379966', 'lastCheck': '8.7', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:59:52,668::task::1185::TaskManager.Task::(prepare) Task=`9e19003d-90b9-4a2a-9ed9-da678d55463d`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000485038', 'lastCheck': '9.2', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000346167', 'lastCheck': '9.3', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.6', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00051056', 'lastCheck': '8.8', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.6', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '141.6', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000379966', 'lastCheck': '8.7', 'valid': True}} >Thread-14::DEBUG::2014-04-11 22:59:52,668::task::595::TaskManager.Task::(_updateState) Task=`9e19003d-90b9-4a2a-9ed9-da678d55463d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:52,668::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:52,669::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:52,669::task::990::TaskManager.Task::(_decref) Task=`9e19003d-90b9-4a2a-9ed9-da678d55463d`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:53,170::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:53,171::task::595::TaskManager.Task::(_updateState) Task=`f09ca30a-efff-4a7e-8573-75520490eae2`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:53,171::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:53,172::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:53,172::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:53,172::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:53,173::task::1185::TaskManager.Task::(prepare) Task=`f09ca30a-efff-4a7e-8573-75520490eae2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:53,173::task::595::TaskManager.Task::(_updateState) Task=`f09ca30a-efff-4a7e-8573-75520490eae2`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:53,173::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:53,173::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:53,174::task::990::TaskManager.Task::(_decref) Task=`f09ca30a-efff-4a7e-8573-75520490eae2`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 22:59:53,340::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 22:59:53,354::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000327478 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 22:59:53,508::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 22:59:53,521::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.00051355 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 22:59:53,893::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 22:59:53,905::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000341264 s, 2.2 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 22:59:53,952::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 22:59:53,964::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000320057 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 22:59:54,179::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:54,180::task::595::TaskManager.Task::(_updateState) Task=`6853ab5b-19bd-4cda-8a10-e812a4dc000a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:54,181::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:54,181::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:54,181::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:54,182::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:54,182::task::1185::TaskManager.Task::(prepare) Task=`6853ab5b-19bd-4cda-8a10-e812a4dc000a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:54,183::task::595::TaskManager.Task::(_updateState) Task=`6853ab5b-19bd-4cda-8a10-e812a4dc000a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:54,183::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:54,183::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:54,184::task::990::TaskManager.Task::(_decref) Task=`6853ab5b-19bd-4cda-8a10-e812a4dc000a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:55,189::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:55,189::task::595::TaskManager.Task::(_updateState) Task=`756ccd38-d47b-4ab7-bcfd-43db5b3a56f8`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:55,190::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:55,190::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:55,190::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:55,191::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:55,191::task::1185::TaskManager.Task::(prepare) Task=`756ccd38-d47b-4ab7-bcfd-43db5b3a56f8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:55,191::task::595::TaskManager.Task::(_updateState) Task=`756ccd38-d47b-4ab7-bcfd-43db5b3a56f8`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:55,191::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:55,192::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:55,192::task::990::TaskManager.Task::(_decref) Task=`756ccd38-d47b-4ab7-bcfd-43db5b3a56f8`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:56,196::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:56,197::task::595::TaskManager.Task::(_updateState) Task=`56c75602-64e2-445b-88aa-239c73ec458d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:56,197::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:56,197::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:56,198::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:56,198::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:56,198::task::1185::TaskManager.Task::(prepare) Task=`56c75602-64e2-445b-88aa-239c73ec458d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:56,199::task::595::TaskManager.Task::(_updateState) Task=`56c75602-64e2-445b-88aa-239c73ec458d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:56,199::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:56,199::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:56,199::task::990::TaskManager.Task::(_decref) Task=`56c75602-64e2-445b-88aa-239c73ec458d`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 22:59:57,204::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:57,205::task::595::TaskManager.Task::(_updateState) Task=`d3d31c35-155b-45e8-b88a-bafe62cda340`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:57,205::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:57,205::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:57,205::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:57,206::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:57,206::task::1185::TaskManager.Task::(prepare) Task=`d3d31c35-155b-45e8-b88a-bafe62cda340`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:57,206::task::595::TaskManager.Task::(_updateState) Task=`d3d31c35-155b-45e8-b88a-bafe62cda340`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:57,207::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:57,207::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:57,207::task::990::TaskManager.Task::(_decref) Task=`d3d31c35-155b-45e8-b88a-bafe62cda340`::ref 0 aborting False >Thread-33::ERROR::2014-04-11 22:59:58,085::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 22:59:58,086::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-39::ERROR::2014-04-11 22:59:58,086::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-33::DEBUG::2014-04-11 22:59:58,095::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::ERROR::2014-04-11 22:59:58,096::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-33::DEBUG::2014-04-11 22:59:58,097::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::DEBUG::2014-04-11 22:59:58,098::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-39::DEBUG::2014-04-11 22:59:58,109::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-14::DEBUG::2014-04-11 22:59:58,213::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:58,214::task::595::TaskManager.Task::(_updateState) Task=`c29661c7-89a4-4bcf-bdf2-eb84c9f28e2f`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:58,214::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:58,215::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:58,215::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:58,215::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:58,216::task::1185::TaskManager.Task::(prepare) Task=`c29661c7-89a4-4bcf-bdf2-eb84c9f28e2f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:58,216::task::595::TaskManager.Task::(_updateState) Task=`c29661c7-89a4-4bcf-bdf2-eb84c9f28e2f`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:58,216::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:58,217::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:58,217::task::990::TaskManager.Task::(_decref) Task=`c29661c7-89a4-4bcf-bdf2-eb84c9f28e2f`::ref 0 aborting False >Thread-33::DEBUG::2014-04-11 22:59:58,379::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 22:59:58,385::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-39::DEBUG::2014-04-11 22:59:58,388::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 22:59:58,402::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::DEBUG::2014-04-11 22:59:58,416::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33::ERROR::2014-04-11 22:59:58,435::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 22:59:58,440::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-39::ERROR::2014-04-11 22:59:58,472::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 22:59:58,472::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-14::DEBUG::2014-04-11 22:59:59,222::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 22:59:59,223::task::595::TaskManager.Task::(_updateState) Task=`0e7992ee-dde7-4f3e-a2cc-938a30e9db87`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 22:59:59,224::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 22:59:59,224::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 22:59:59,224::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 22:59:59,225::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:59,225::task::1185::TaskManager.Task::(prepare) Task=`0e7992ee-dde7-4f3e-a2cc-938a30e9db87`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 22:59:59,225::task::595::TaskManager.Task::(_updateState) Task=`0e7992ee-dde7-4f3e-a2cc-938a30e9db87`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 22:59:59,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 22:59:59,226::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 22:59:59,226::task::990::TaskManager.Task::(_decref) Task=`0e7992ee-dde7-4f3e-a2cc-938a30e9db87`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:00,231::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:00,232::task::595::TaskManager.Task::(_updateState) Task=`ee17572b-fe3b-44fc-8c26-127e11ad87b1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:00,232::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:00,233::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:00,233::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:00,233::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:00,234::task::1185::TaskManager.Task::(prepare) Task=`ee17572b-fe3b-44fc-8c26-127e11ad87b1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:00,234::task::595::TaskManager.Task::(_updateState) Task=`ee17572b-fe3b-44fc-8c26-127e11ad87b1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:00,234::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:00,235::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:00,235::task::990::TaskManager.Task::(_decref) Task=`ee17572b-fe3b-44fc-8c26-127e11ad87b1`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:01,240::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:01,240::task::595::TaskManager.Task::(_updateState) Task=`49f2f467-8685-4f6f-abbb-a675acad16e8`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:01,241::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:01,241::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:01,241::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:01,241::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:01,242::task::1185::TaskManager.Task::(prepare) Task=`49f2f467-8685-4f6f-abbb-a675acad16e8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:01,242::task::595::TaskManager.Task::(_updateState) Task=`49f2f467-8685-4f6f-abbb-a675acad16e8`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:01,242::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:01,243::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:01,243::task::990::TaskManager.Task::(_decref) Task=`49f2f467-8685-4f6f-abbb-a675acad16e8`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:02,248::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:02,248::task::595::TaskManager.Task::(_updateState) Task=`34a529a1-14d2-4ccc-828f-9c67d7be1b59`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:02,249::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:02,249::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:02,249::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:02,250::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:02,250::task::1185::TaskManager.Task::(prepare) Task=`34a529a1-14d2-4ccc-828f-9c67d7be1b59`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:02,250::task::595::TaskManager.Task::(_updateState) Task=`34a529a1-14d2-4ccc-828f-9c67d7be1b59`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:02,251::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:02,251::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:02,251::task::990::TaskManager.Task::(_decref) Task=`34a529a1-14d2-4ccc-828f-9c67d7be1b59`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:03,256::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:03,257::task::595::TaskManager.Task::(_updateState) Task=`ddca807f-0b63-4845-be89-fed7eeb6e8fc`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:03,257::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:03,258::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:03,258::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:03,258::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:03,258::task::1185::TaskManager.Task::(prepare) Task=`ddca807f-0b63-4845-be89-fed7eeb6e8fc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:03,259::task::595::TaskManager.Task::(_updateState) Task=`ddca807f-0b63-4845-be89-fed7eeb6e8fc`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:03,259::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:03,259::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:03,260::task::990::TaskManager.Task::(_decref) Task=`ddca807f-0b63-4845-be89-fed7eeb6e8fc`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 23:00:03,381::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 23:00:03,398::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000343957 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 23:00:03,550::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 23:00:03,563::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000518859 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 23:00:03,935::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 23:00:03,949::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000541906 s, 1.4 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 23:00:03,998::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 23:00:04,010::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.00036034 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 23:00:04,265::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:04,266::task::595::TaskManager.Task::(_updateState) Task=`69a92bdc-708b-4e87-9f82-7dea14d3c9ba`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:04,267::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:04,267::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:04,267::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:04,268::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:04,268::task::1185::TaskManager.Task::(prepare) Task=`69a92bdc-708b-4e87-9f82-7dea14d3c9ba`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:04,268::task::595::TaskManager.Task::(_updateState) Task=`69a92bdc-708b-4e87-9f82-7dea14d3c9ba`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:04,269::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:04,269::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:04,269::task::990::TaskManager.Task::(_decref) Task=`69a92bdc-708b-4e87-9f82-7dea14d3c9ba`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:05,274::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:05,275::task::595::TaskManager.Task::(_updateState) Task=`36f5c837-a6fa-44fa-8645-c9351ea14568`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:05,275::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:05,276::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:05,276::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:05,276::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:05,277::task::1185::TaskManager.Task::(prepare) Task=`36f5c837-a6fa-44fa-8645-c9351ea14568`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:05,277::task::595::TaskManager.Task::(_updateState) Task=`36f5c837-a6fa-44fa-8645-c9351ea14568`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:05,277::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:05,278::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:05,278::task::990::TaskManager.Task::(_decref) Task=`36f5c837-a6fa-44fa-8645-c9351ea14568`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:06,283::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:06,283::task::595::TaskManager.Task::(_updateState) Task=`d16d726e-320b-44de-892c-ebba8afc5653`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:06,284::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:06,284::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:06,284::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:06,285::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:06,285::task::1185::TaskManager.Task::(prepare) Task=`d16d726e-320b-44de-892c-ebba8afc5653`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:06,285::task::595::TaskManager.Task::(_updateState) Task=`d16d726e-320b-44de-892c-ebba8afc5653`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:06,286::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:06,286::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:06,286::task::990::TaskManager.Task::(_decref) Task=`d16d726e-320b-44de-892c-ebba8afc5653`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:07,291::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:07,309::task::595::TaskManager.Task::(_updateState) Task=`a3cfe5ef-41bf-466f-8d68-ea0bf9b7d280`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:07,309::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:07,310::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:07,310::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:07,310::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:07,310::task::1185::TaskManager.Task::(prepare) Task=`a3cfe5ef-41bf-466f-8d68-ea0bf9b7d280`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:07,311::task::595::TaskManager.Task::(_updateState) Task=`a3cfe5ef-41bf-466f-8d68-ea0bf9b7d280`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:07,311::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:07,311::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:07,312::task::990::TaskManager.Task::(_decref) Task=`a3cfe5ef-41bf-466f-8d68-ea0bf9b7d280`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:07,983::task::595::TaskManager.Task::(_updateState) Task=`390e045a-9112-4ff3-bb06-6183abd87b5b`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:07,983::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 23:00:07,984::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000518859', 'lastCheck': '4.4', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000343957', 'lastCheck': '4.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.5', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000541906', 'lastCheck': '4.0', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '156.9', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00036034', 'lastCheck': '4.0', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:07,984::task::1185::TaskManager.Task::(prepare) Task=`390e045a-9112-4ff3-bb06-6183abd87b5b`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000518859', 'lastCheck': '4.4', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000343957', 'lastCheck': '4.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.5', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000541906', 'lastCheck': '4.0', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.5', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000570109', 'lastCheck': '156.9', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.00036034', 'lastCheck': '4.0', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:07,985::task::595::TaskManager.Task::(_updateState) Task=`390e045a-9112-4ff3-bb06-6183abd87b5b`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:07,985::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:07,985::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:07,986::task::990::TaskManager.Task::(_decref) Task=`390e045a-9112-4ff3-bb06-6183abd87b5b`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:08,316::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:08,317::task::595::TaskManager.Task::(_updateState) Task=`1741d324-ae09-40d0-95ab-fdffd3e7f538`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:08,317::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:08,317::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:08,318::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:08,318::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:08,318::task::1185::TaskManager.Task::(prepare) Task=`1741d324-ae09-40d0-95ab-fdffd3e7f538`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:08,319::task::595::TaskManager.Task::(_updateState) Task=`1741d324-ae09-40d0-95ab-fdffd3e7f538`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:08,319::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:08,319::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:08,320::task::990::TaskManager.Task::(_decref) Task=`1741d324-ae09-40d0-95ab-fdffd3e7f538`::ref 0 aborting False >Thread-33::ERROR::2014-04-11 23:00:08,454::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 23:00:08,455::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 23:00:08,455::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33::DEBUG::2014-04-11 23:00:08,456::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::ERROR::2014-04-11 23:00:08,474::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 23:00:08,475::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 23:00:08,475::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-39::DEBUG::2014-04-11 23:00:08,476::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::DEBUG::2014-04-11 23:00:08,741::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 23:00:08,744::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::ERROR::2014-04-11 23:00:08,782::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-39::DEBUG::2014-04-11 23:00:08,783::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-33::ERROR::2014-04-11 23:00:08,786::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-39::WARNING::2014-04-11 23:00:08,792::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::DEBUG::2014-04-11 23:00:08,793::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39::ERROR::2014-04-11 23:00:08,823::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 23:00:08,824::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-14::DEBUG::2014-04-11 23:00:09,325::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:09,326::task::595::TaskManager.Task::(_updateState) Task=`4adc83bc-3a72-4cd9-a843-4bf622cd7eaa`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:09,326::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:09,327::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:09,327::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:09,328::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:09,328::task::1185::TaskManager.Task::(prepare) Task=`4adc83bc-3a72-4cd9-a843-4bf622cd7eaa`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:09,328::task::595::TaskManager.Task::(_updateState) Task=`4adc83bc-3a72-4cd9-a843-4bf622cd7eaa`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:09,329::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:09,329::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:09,329::task::990::TaskManager.Task::(_decref) Task=`4adc83bc-3a72-4cd9-a843-4bf622cd7eaa`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:10,334::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:10,335::task::595::TaskManager.Task::(_updateState) Task=`c1382639-ecb5-4d5d-8318-64a9d652f897`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:10,335::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:10,336::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:10,336::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:10,336::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:10,336::task::1185::TaskManager.Task::(prepare) Task=`c1382639-ecb5-4d5d-8318-64a9d652f897`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:10,337::task::595::TaskManager.Task::(_updateState) Task=`c1382639-ecb5-4d5d-8318-64a9d652f897`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:10,337::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:10,337::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:10,338::task::990::TaskManager.Task::(_decref) Task=`c1382639-ecb5-4d5d-8318-64a9d652f897`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:11,342::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:11,342::task::595::TaskManager.Task::(_updateState) Task=`5ef7a2b0-85c0-4b1c-bfad-e0ebfefa0efc`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:11,343::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:11,343::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:11,343::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:11,344::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:11,344::task::1185::TaskManager.Task::(prepare) Task=`5ef7a2b0-85c0-4b1c-bfad-e0ebfefa0efc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:11,344::task::595::TaskManager.Task::(_updateState) Task=`5ef7a2b0-85c0-4b1c-bfad-e0ebfefa0efc`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:11,345::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:11,345::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:11,345::task::990::TaskManager.Task::(_decref) Task=`5ef7a2b0-85c0-4b1c-bfad-e0ebfefa0efc`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:12,349::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:12,350::task::595::TaskManager.Task::(_updateState) Task=`8c4048c4-8e95-41e7-9c0c-8893aaa1e9d3`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:12,350::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:12,351::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:12,351::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:12,351::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:12,352::task::1185::TaskManager.Task::(prepare) Task=`8c4048c4-8e95-41e7-9c0c-8893aaa1e9d3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:12,352::task::595::TaskManager.Task::(_updateState) Task=`8c4048c4-8e95-41e7-9c0c-8893aaa1e9d3`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:12,352::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:12,352::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:12,353::task::990::TaskManager.Task::(_decref) Task=`8c4048c4-8e95-41e7-9c0c-8893aaa1e9d3`::ref 0 aborting False >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,624::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:12,625::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,677::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,677::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,693::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,694::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,694::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,695::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,735::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,735::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`2ee18c70-c625-479f-81c3-1b070c2f8207`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,736::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,736::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,737::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`2ee18c70-c625-479f-81c3-1b070c2f8207`::Granted request >9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:12,737::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a` >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,738::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`59a48651-840c-4a4f-951b-ba07b08418b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,738::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,739::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,739::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`59a48651-840c-4a4f-951b-ba07b08418b4`::Granted request >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,754::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,756::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,772::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,774::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:12,774::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered >9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:12,775::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,775::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,775::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,776::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,776::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,776::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,791::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,792::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,792::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >Thread-342::DEBUG::2014-04-11 23:00:12,793::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-343::DEBUG::2014-04-11 23:00:12,794::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-342::DEBUG::2014-04-11 23:00:12,795::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`d1b465b3-aa93-4fbc-9e01-29b5137037c1`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-344::DEBUG::2014-04-11 23:00:12,795::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-343::DEBUG::2014-04-11 23:00:12,797::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`4a2d70c9-ad7f-4e76-8f8a-fcf08802bacc`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-345::ERROR::2014-04-11 23:00:12,797::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-342::DEBUG::2014-04-11 23:00:12,799::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive' >Thread-346::ERROR::2014-04-11 23:00:12,799::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-344::DEBUG::2014-04-11 23:00:12,800::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`41ff77b2-310a-40a2-94bd-071757ba5a3b`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-343::DEBUG::2014-04-11 23:00:12,800::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive' >Thread-345::ERROR::2014-04-11 23:00:12,801::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-347::DEBUG::2014-04-11 23:00:12,801::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 21619c8e-99ea-4813-be02-d708971e5393 >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,802::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-342::DEBUG::2014-04-11 23:00:12,802::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user) >Thread-346::ERROR::2014-04-11 23:00:12,802::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-344::DEBUG::2014-04-11 23:00:12,803::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive' >Thread-345::DEBUG::2014-04-11 23:00:12,804::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-347::DEBUG::2014-04-11 23:00:12,805::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`8cd2444d-b94a-42a5-9d29-3e26eac098f0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-342::DEBUG::2014-04-11 23:00:12,806::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`d1b465b3-aa93-4fbc-9e01-29b5137037c1`::Granted request >Thread-345::DEBUG::2014-04-11 23:00:12,809::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-346::DEBUG::2014-04-11 23:00:12,809::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-347::DEBUG::2014-04-11 23:00:12,810::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive' >Thread-343::DEBUG::2014-04-11 23:00:12,810::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user) >Thread-342::DEBUG::2014-04-11 23:00:12,823::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`b0f5bd84-921a-42a1-b858-7cd3acebd1fe`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-346::DEBUG::2014-04-11 23:00:12,825::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-343::DEBUG::2014-04-11 23:00:12,827::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`4a2d70c9-ad7f-4e76-8f8a-fcf08802bacc`::Granted request >Thread-342::DEBUG::2014-04-11 23:00:12,827::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,839::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-343::DEBUG::2014-04-11 23:00:12,842::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`2f87ab4b-0ec3-4813-b7d3-3b2543939062`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,842::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-343::DEBUG::2014-04-11 23:00:12,843::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,843::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-344::DEBUG::2014-04-11 23:00:12,844::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user) >Thread-344::DEBUG::2014-04-11 23:00:12,846::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`41ff77b2-310a-40a2-94bd-071757ba5a3b`::Granted request >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,847::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms >Thread-347::DEBUG::2014-04-11 23:00:12,847::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user) >Thread-344::DEBUG::2014-04-11 23:00:12,850::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`371c311f-b46b-4fde-a322-1ee4982d7eb4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-347::DEBUG::2014-04-11 23:00:12,850::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`8cd2444d-b94a-42a5-9d29-3e26eac098f0`::Granted request >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,851::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks >Thread-344::DEBUG::2014-04-11 23:00:12,851::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive' >Thread-342::DEBUG::2014-04-11 23:00:12,852::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user) >Thread-347::DEBUG::2014-04-11 23:00:12,855::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`213cfa65-acb1-4cf4-9871-4e4fe811bb04`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-342::DEBUG::2014-04-11 23:00:12,855::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`b0f5bd84-921a-42a1-b858-7cd3acebd1fe`::Granted request >Thread-347::DEBUG::2014-04-11 23:00:12,856::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive' >Thread-343::DEBUG::2014-04-11 23:00:12,857::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user) >Thread-343::DEBUG::2014-04-11 23:00:12,859::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`2f87ab4b-0ec3-4813-b7d3-3b2543939062`::Granted request >Thread-344::DEBUG::2014-04-11 23:00:12,859::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user) >Thread-343::DEBUG::2014-04-11 23:00:12,860::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` is not a data domain it is an 2 domain, not upgrading >Thread-344::DEBUG::2014-04-11 23:00:12,860::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`371c311f-b46b-4fde-a322-1ee4982d7eb4`::Granted request >Thread-343::DEBUG::2014-04-11 23:00:12,861::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' >Thread-347::DEBUG::2014-04-11 23:00:12,862::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user) >Thread-347::DEBUG::2014-04-11 23:00:12,864::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`213cfa65-acb1-4cf4-9871-4e4fe811bb04`::Granted request >Thread-343::DEBUG::2014-04-11 23:00:12,868::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users) >Thread-343::DEBUG::2014-04-11 23:00:12,869::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it. >Thread-343::DEBUG::2014-04-11 23:00:12,869::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records. >Thread-343::DEBUG::2014-04-11 23:00:12,870::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' >Thread-343::DEBUG::2014-04-11 23:00:12,870::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users) >Thread-343::DEBUG::2014-04-11 23:00:12,871::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it. >Thread-343::DEBUG::2014-04-11 23:00:12,871::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:12,872::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >Thread-344::DEBUG::2014-04-11 23:00:12,881::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40'] >Thread-344::DEBUG::2014-04-11 23:00:12,881::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' >Thread-344::DEBUG::2014-04-11 23:00:12,882::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users) >Thread-344::DEBUG::2014-04-11 23:00:12,882::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it. >Thread-344::DEBUG::2014-04-11 23:00:12,882::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records. >Thread-344::DEBUG::2014-04-11 23:00:12,883::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' >Thread-344::DEBUG::2014-04-11 23:00:12,883::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users) >Thread-344::DEBUG::2014-04-11 23:00:12,883::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it. >Thread-344::DEBUG::2014-04-11 23:00:12,884::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records. >Thread-342::DEBUG::2014-04-11 23:00:12,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955'] >Thread-342::DEBUG::2014-04-11 23:00:12,889::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-342::DEBUG::2014-04-11 23:00:12,889::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-342::DEBUG::2014-04-11 23:00:12,890::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-342::DEBUG::2014-04-11 23:00:12,890::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-342::DEBUG::2014-04-11 23:00:12,890::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-342::DEBUG::2014-04-11 23:00:12,891::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-342::DEBUG::2014-04-11 23:00:12,891::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-342::DEBUG::2014-04-11 23:00:12,891::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-347::DEBUG::2014-04-11 23:00:12,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2'] >Thread-347::DEBUG::2014-04-11 23:00:12,895::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' >Thread-347::DEBUG::2014-04-11 23:00:12,896::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' (0 active users) >Thread-347::DEBUG::2014-04-11 23:00:12,896::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it. >Thread-347::DEBUG::2014-04-11 23:00:12,896::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393', Clearing records. >Thread-347::DEBUG::2014-04-11 23:00:12,897::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' >Thread-347::DEBUG::2014-04-11 23:00:12,897::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' (0 active users) >Thread-347::DEBUG::2014-04-11 23:00:12,897::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it. >Thread-347::DEBUG::2014-04-11 23:00:12,897::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393', Clearing records. >Thread-345::DEBUG::2014-04-11 23:00:13,100::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-345::WARNING::2014-04-11 23:00:13,104::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-346::DEBUG::2014-04-11 23:00:13,143::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-345::ERROR::2014-04-11 23:00:13,143::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,161::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,164::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-346::WARNING::2014-04-11 23:00:13,165::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,167::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,192::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-346::ERROR::2014-04-11 23:00:13,229::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-14::DEBUG::2014-04-11 23:00:13,358::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:13,359::task::595::TaskManager.Task::(_updateState) Task=`b25a3033-503b-4b1f-bb94-79cd05384430`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:13,360::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:13,360::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:13,360::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:13,361::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:13,361::task::1185::TaskManager.Task::(prepare) Task=`b25a3033-503b-4b1f-bb94-79cd05384430`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:13,361::task::595::TaskManager.Task::(_updateState) Task=`b25a3033-503b-4b1f-bb94-79cd05384430`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:13,362::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:13,362::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:13,362::task::990::TaskManager.Task::(_decref) Task=`b25a3033-503b-4b1f-bb94-79cd05384430`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 23:00:13,426::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 23:00:13,439::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000322894 s, 1.1 MB/s\n'; <rc> = 0 >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,476::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >9d3d90f5-8732-4a54-854e-573f3fdcd2de::WARNING::2014-04-11 23:00:13,478::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,479::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,510::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,510::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,526::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,526::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,526::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,528::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-22::DEBUG::2014-04-11 23:00:13,596::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 23:00:13,610::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000512307 s, 1.4 MB/s\n'; <rc> = 0 >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,796::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >9d3d90f5-8732-4a54-854e-573f3fdcd2de::WARNING::2014-04-11 23:00:13,798::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,798::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,832::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,833::sp::329::Storage.StoragePool::(startSpm) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm > self._updateDomainsRole() > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole > domain = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,833::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,834::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,835::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d69c0c5d-efe3-40da-8172-8c5758643b61`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,835::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,835::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,836::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d69c0c5d-efe3-40da-8172-8c5758643b61`::Granted request >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,837::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`9236862c-e98a-44dc-86a1-29e2de279820`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,837::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' for lock type 'exclusive' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,838::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free. Now locking as 'exclusive' (1 active user) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,838::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`9236862c-e98a-44dc-86a1-29e2de279820`::Granted request >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,839::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,839::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' (0 active users) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,839::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free, finding out if anyone is waiting for it. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,840::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb', Clearing records. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,840::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`6447f156-d9eb-4db9-ad58-da3049474424`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,841::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' for lock type 'exclusive' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,841::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free. Now locking as 'exclusive' (1 active user) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,841::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`6447f156-d9eb-4db9-ad58-da3049474424`::Granted request >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,842::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,842::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' (0 active users) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,843::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free, finding out if anyone is waiting for it. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,843::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48', Clearing records. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,844::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,844::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,844::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,845::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,845::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,846::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,847::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,856::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >9d3d90f5-8732-4a54-854e-573f3fdcd2de::INFO::2014-04-11 23:00:13,857::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,859::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released >9d3d90f5-8732-4a54-854e-573f3fdcd2de::ERROR::2014-04-11 23:00:13,860::task::866::TaskManager.Task::(_setError) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/storage/task.py", line 334, in run > return self.cmd(*self.argslist, **self.argsdict) > File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm > self._updateDomainsRole() > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole > domain = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,862::task::885::TaskManager.Task::(_run) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Task._run: 9d3d90f5-8732-4a54-854e-573f3fdcd2de () {} failed - stopping task >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,863::task::1211::TaskManager.Task::(stop) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::stopping in state running (force False) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,863::task::990::TaskManager.Task::(_decref) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::ref 1 aborting True >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,863::task::916::TaskManager.Task::(_runJobs) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,864::task::990::TaskManager.Task::(_decref) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::ref 0 aborting True >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,864::task::925::TaskManager.Task::(_doAbort) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Task._doAbort: force False >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,864::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,865::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state running -> state aborting >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,865::task::550::TaskManager.Task::(__state_aborting) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::_aborting: recover policy auto >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,865::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state aborting -> state racquiring >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,866::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state racquiring -> state recovering >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,866::task::798::TaskManager.Task::(_recover) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::_recover >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,866::task::805::TaskManager.Task::(_recover) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::running recovery None >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,867::task::786::TaskManager.Task::(_recoverDone) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::Recover Done: state recovering >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,867::task::595::TaskManager.Task::(_updateState) Task=`9d3d90f5-8732-4a54-854e-573f3fdcd2de`::moving from state recovering -> state recovered >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,867::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>} >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,868::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,868::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,868::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,869::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,869::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:13,870::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-29::DEBUG::2014-04-11 23:00:13,991::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 23:00:14,007::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000459157 s, 1.6 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 23:00:14,042::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 23:00:14,053::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000280847 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 23:00:14,368::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:14,369::task::595::TaskManager.Task::(_updateState) Task=`84ce285c-fb07-4c63-abe2-f9958168868e`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:14,369::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:14,369::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:14,370::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'} >Thread-14::INFO::2014-04-11 23:00:14,370::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:14,370::task::1185::TaskManager.Task::(prepare) Task=`84ce285c-fb07-4c63-abe2-f9958168868e`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9d3d90f5-8732-4a54-854e-573f3fdcd2de'}} >Thread-14::DEBUG::2014-04-11 23:00:14,371::task::595::TaskManager.Task::(_updateState) Task=`84ce285c-fb07-4c63-abe2-f9958168868e`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:14,371::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:14,371::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:14,372::task::990::TaskManager.Task::(_decref) Task=`84ce285c-fb07-4c63-abe2-f9958168868e`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:14,382::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:14,383::task::595::TaskManager.Task::(_updateState) Task=`8846179f-ed46-4bb2-95ec-cafb62b1af01`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:14,384::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:14,399::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:14,399::task::1185::TaskManager.Task::(prepare) Task=`8846179f-ed46-4bb2-95ec-cafb62b1af01`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:14,400::task::595::TaskManager.Task::(_updateState) Task=`8846179f-ed46-4bb2-95ec-cafb62b1af01`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:14,400::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:14,400::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:14,401::task::990::TaskManager.Task::(_decref) Task=`8846179f-ed46-4bb2-95ec-cafb62b1af01`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:14,482::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:14,483::task::595::TaskManager.Task::(_updateState) Task=`c2fa8758-45e1-4745-9d24-5d1e1a268fc7`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:14,483::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='9d3d90f5-8732-4a54-854e-573f3fdcd2de', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:14,484::taskManager::161::TaskManager::(clearTask) Entry. taskID: 9d3d90f5-8732-4a54-854e-573f3fdcd2de >Thread-14::DEBUG::2014-04-11 23:00:14,484::taskManager::166::TaskManager::(clearTask) Return. >Thread-14::INFO::2014-04-11 23:00:14,484::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-14::DEBUG::2014-04-11 23:00:14,485::task::1185::TaskManager.Task::(prepare) Task=`c2fa8758-45e1-4745-9d24-5d1e1a268fc7`::finished: None >Thread-14::DEBUG::2014-04-11 23:00:14,485::task::595::TaskManager.Task::(_updateState) Task=`c2fa8758-45e1-4745-9d24-5d1e1a268fc7`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:14,485::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:14,486::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:14,486::task::990::TaskManager.Task::(_decref) Task=`c2fa8758-45e1-4745-9d24-5d1e1a268fc7`::ref 0 aborting False >Thread-33::ERROR::2014-04-11 23:00:18,792::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 23:00:18,792::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 23:00:18,793::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33::DEBUG::2014-04-11 23:00:18,793::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::ERROR::2014-04-11 23:00:18,825::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 23:00:18,826::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 23:00:18,827::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-39::DEBUG::2014-04-11 23:00:18,828::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::DEBUG::2014-04-11 23:00:19,066::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 23:00:19,068::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::ERROR::2014-04-11 23:00:19,110::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 23:00:19,115::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-39::DEBUG::2014-04-11 23:00:19,128::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 23:00:19,134::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::DEBUG::2014-04-11 23:00:19,137::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39::ERROR::2014-04-11 23:00:19,184::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 23:00:19,185::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-43::DEBUG::2014-04-11 23:00:22,646::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 23:00:22,660::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000500292 s, 1.5 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 23:00:23,303::task::595::TaskManager.Task::(_updateState) Task=`de82191d-a9af-4594-a20e-b253eb38a2e0`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:23,303::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 23:00:23,304::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000512307', 'lastCheck': '9.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000322894', 'lastCheck': '9.9', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000459157', 'lastCheck': '9.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000500292', 'lastCheck': '0.6', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000280847', 'lastCheck': '9.2', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:23,305::task::1185::TaskManager.Task::(prepare) Task=`de82191d-a9af-4594-a20e-b253eb38a2e0`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000512307', 'lastCheck': '9.7', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000322894', 'lastCheck': '9.9', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000459157', 'lastCheck': '9.3', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000500292', 'lastCheck': '0.6', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000280847', 'lastCheck': '9.2', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:23,305::task::595::TaskManager.Task::(_updateState) Task=`de82191d-a9af-4594-a20e-b253eb38a2e0`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:23,305::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:23,306::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:23,306::task::990::TaskManager.Task::(_decref) Task=`de82191d-a9af-4594-a20e-b253eb38a2e0`::ref 0 aborting False >Thread-24::DEBUG::2014-04-11 23:00:23,466::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 23:00:23,480::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.00032068 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 23:00:23,638::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 23:00:23,652::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000575685 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 23:00:24,035::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 23:00:24,048::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000384216 s, 2.0 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 23:00:24,083::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 23:00:24,094::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000271965 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 23:00:24,642::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:24,644::task::595::TaskManager.Task::(_updateState) Task=`10ab6b4e-ce22-44c8-986e-a0d98914e5e2`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:24,644::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:24,659::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:24,659::task::1185::TaskManager.Task::(prepare) Task=`10ab6b4e-ce22-44c8-986e-a0d98914e5e2`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:24,660::task::595::TaskManager.Task::(_updateState) Task=`10ab6b4e-ce22-44c8-986e-a0d98914e5e2`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:24,660::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:24,660::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:24,661::task::990::TaskManager.Task::(_decref) Task=`10ab6b4e-ce22-44c8-986e-a0d98914e5e2`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:24,665::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:24,666::task::595::TaskManager.Task::(_updateState) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:24,666::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-14::ERROR::2014-04-11 23:00:24,667::task::866::TaskManager.Task::(_setError) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2115, in getAllTasksStatuses > allTasksStatus = sp.getAllTasksStatuses() > File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper > raise SecureError("Secured object is not in safe state") >SecureError: Secured object is not in safe state >Thread-14::DEBUG::2014-04-11 23:00:24,667::task::885::TaskManager.Task::(_run) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::Task._run: 4905fe14-b30b-45f2-aee8-fd9523604d46 () {} failed - stopping task >Thread-14::DEBUG::2014-04-11 23:00:24,668::task::1211::TaskManager.Task::(stop) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 23:00:24,668::task::990::TaskManager.Task::(_decref) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::ref 1 aborting True >Thread-14::INFO::2014-04-11 23:00:24,668::task::1168::TaskManager.Task::(prepare) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100 >Thread-14::DEBUG::2014-04-11 23:00:24,668::task::1173::TaskManager.Task::(prepare) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::Prepare: aborted: Secured object is not in safe state >Thread-14::DEBUG::2014-04-11 23:00:24,669::task::990::TaskManager.Task::(_decref) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 23:00:24,669::task::925::TaskManager.Task::(_doAbort) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 23:00:24,669::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:24,670::task::595::TaskManager.Task::(_updateState) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 23:00:24,670::task::550::TaskManager.Task::(__state_aborting) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 23:00:24,670::task::595::TaskManager.Task::(_updateState) Task=`4905fe14-b30b-45f2-aee8-fd9523604d46`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 23:00:24,670::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:24,671::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 23:00:24,671::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SecureError: Secured object is not in safe state >Thread-14::DEBUG::2014-04-11 23:00:24,788::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:24,789::task::595::TaskManager.Task::(_updateState) Task=`2f94f4ba-858e-4021-8aa8-e7a5cf0cec2a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:24,789::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:24,802::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:24,802::task::1185::TaskManager.Task::(prepare) Task=`2f94f4ba-858e-4021-8aa8-e7a5cf0cec2a`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:24,802::task::595::TaskManager.Task::(_updateState) Task=`2f94f4ba-858e-4021-8aa8-e7a5cf0cec2a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:24,802::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:24,803::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:24,803::task::990::TaskManager.Task::(_decref) Task=`2f94f4ba-858e-4021-8aa8-e7a5cf0cec2a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:24,850::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:24,850::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:24,851::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None) >Thread-14::DEBUG::2014-04-11 23:00:24,852::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`f9634649-0ce2-4e05-8639-a00b6507d0ad`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart' >Thread-14::DEBUG::2014-04-11 23:00:24,852::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 23:00:24,853::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 23:00:24,853::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`f9634649-0ce2-4e05-8639-a00b6507d0ad`::Granted request >Thread-14::DEBUG::2014-04-11 23:00:24,854::task::827::TaskManager.Task::(resourceAcquired) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive) >Thread-14::DEBUG::2014-04-11 23:00:24,854::task::990::TaskManager.Task::(_decref) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::ref 1 aborting False >Thread-14::INFO::2014-04-11 23:00:24,854::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-14::DEBUG::2014-04-11 23:00:24,855::task::1180::TaskManager.Task::(prepare) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Prepare: 1 jobs exist, move to acquiring >Thread-14::DEBUG::2014-04-11 23:00:24,855::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state preparing -> state acquiring >Thread-14::DEBUG::2014-04-11 23:00:24,855::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state acquiring -> state queued >Thread-14::DEBUG::2014-04-11 23:00:24,856::taskManager::50::TaskManager::(_queueTask) queuing task: 0a360c1f-47b6-41f6-a72e-e92a32263e98 >Thread-14::DEBUG::2014-04-11 23:00:24,856::taskManager::56::TaskManager::(_queueTask) task queued: 0a360c1f-47b6-41f6-a72e-e92a32263e98 >9d3d90f5-8732-4a54-854e-573f3fdcd2de::DEBUG::2014-04-11 23:00:24,856::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-14::DEBUG::2014-04-11 23:00:24,857::task::1182::TaskManager.Task::(prepare) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::returning >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,857::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 0a360c1f-47b6-41f6-a72e-e92a32263e98 running: <bound method Task.commit of <storage.task.Task instance at 0x2e38878>> with: None >Thread-14::DEBUG::2014-04-11 23:00:24,858::task::990::TaskManager.Task::(_decref) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::ref 0 aborting False >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,858::task::1194::TaskManager.Task::(commit) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::committing task: 0a360c1f-47b6-41f6-a72e-e92a32263e98 >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,859::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state queued -> state running >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,860::task::905::TaskManager.Task::(_runJobs) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,861::task::331::TaskManager.Task::(run) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,861::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,861::misc::764::SamplingMethod::(__call__) Got in to sampling method >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,878::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,879::misc::772::SamplingMethod::(__call__) Returning last result >0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:24,892::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,893::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:24,893::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,931::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,931::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,947::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,947::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,948::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,948::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,963::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,964::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`f264ec2c-72a7-4b4b-aaae-d22e90555b03`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,964::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,964::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,965::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`f264ec2c-72a7-4b4b-aaae-d22e90555b03`::Granted request >0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:24,965::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a` >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,966::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`2580c927-cd8e-44d0-b401-aeaa7122bd01`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,966::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,967::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,967::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`2580c927-cd8e-44d0-b401-aeaa7122bd01`::Granted request >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,983::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:24,984::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,000::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,002::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:25,002::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered >0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:25,003::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,003::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,003::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,004::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,004::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,004::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,020::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,020::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >Thread-390::DEBUG::2014-04-11 23:00:25,021::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-391::DEBUG::2014-04-11 23:00:25,022::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-390::DEBUG::2014-04-11 23:00:25,023::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`9029a627-c541-4cb3-b616-b56015928f9d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-392::DEBUG::2014-04-11 23:00:25,024::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-390::DEBUG::2014-04-11 23:00:25,024::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive' >Thread-391::DEBUG::2014-04-11 23:00:25,025::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`c8eec928-2658-460b-a41e-7859faf96690`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-393::ERROR::2014-04-11 23:00:25,026::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-392::DEBUG::2014-04-11 23:00:25,026::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`c8efee26-1822-410e-9d26-e32186b6241f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-394::ERROR::2014-04-11 23:00:25,028::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-390::DEBUG::2014-04-11 23:00:25,028::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user) >Thread-391::DEBUG::2014-04-11 23:00:25,029::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive' >Thread-395::DEBUG::2014-04-11 23:00:25,029::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 21619c8e-99ea-4813-be02-d708971e5393 >Thread-393::ERROR::2014-04-11 23:00:25,030::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,030::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-392::DEBUG::2014-04-11 23:00:25,031::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive' >Thread-394::ERROR::2014-04-11 23:00:25,031::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-390::DEBUG::2014-04-11 23:00:25,031::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`9029a627-c541-4cb3-b616-b56015928f9d`::Granted request >Thread-395::DEBUG::2014-04-11 23:00:25,033::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`a672482e-7fbe-40fd-bcfb-429da3fb499c`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-393::DEBUG::2014-04-11 23:00:25,033::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-391::DEBUG::2014-04-11 23:00:25,035::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user) >Thread-395::DEBUG::2014-04-11 23:00:25,035::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive' >Thread-390::DEBUG::2014-04-11 23:00:25,036::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`4bc51ac8-a85d-42a5-b0b8-629458b60845`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-393::DEBUG::2014-04-11 23:00:25,037::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-394::DEBUG::2014-04-11 23:00:25,038::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-391::DEBUG::2014-04-11 23:00:25,038::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`c8eec928-2658-460b-a41e-7859faf96690`::Granted request >Thread-390::DEBUG::2014-04-11 23:00:25,039::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive' >Thread-394::DEBUG::2014-04-11 23:00:25,053::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,054::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-391::DEBUG::2014-04-11 23:00:25,057::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`e1eb6b89-c2d4-4754-b833-c0ccbc551bdd`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,057::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-391::DEBUG::2014-04-11 23:00:25,068::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,069::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-392::DEBUG::2014-04-11 23:00:25,071::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user) >Thread-392::DEBUG::2014-04-11 23:00:25,071::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`c8efee26-1822-410e-9d26-e32186b6241f`::Granted request >Thread-395::DEBUG::2014-04-11 23:00:25,076::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,077::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms >Thread-395::DEBUG::2014-04-11 23:00:25,078::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`a672482e-7fbe-40fd-bcfb-429da3fb499c`::Granted request >Thread-392::DEBUG::2014-04-11 23:00:25,079::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`daa390c2-033f-45ff-9716-a3535fbe9d58`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-390::DEBUG::2014-04-11 23:00:25,080::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,081::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks >Thread-395::DEBUG::2014-04-11 23:00:25,082::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`5be6d11d-cbb2-421b-a786-4d1628a24236`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-392::DEBUG::2014-04-11 23:00:25,082::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive' >Thread-390::DEBUG::2014-04-11 23:00:25,083::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`4bc51ac8-a85d-42a5-b0b8-629458b60845`::Granted request >Thread-395::DEBUG::2014-04-11 23:00:25,084::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive' >Thread-391::DEBUG::2014-04-11 23:00:25,086::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user) >Thread-391::DEBUG::2014-04-11 23:00:25,088::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`e1eb6b89-c2d4-4754-b833-c0ccbc551bdd`::Granted request >Thread-392::DEBUG::2014-04-11 23:00:25,089::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user) >Thread-391::DEBUG::2014-04-11 23:00:25,090::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` is not a data domain it is an 2 domain, not upgrading >Thread-392::DEBUG::2014-04-11 23:00:25,090::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`daa390c2-033f-45ff-9716-a3535fbe9d58`::Granted request >Thread-391::DEBUG::2014-04-11 23:00:25,090::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' >Thread-395::DEBUG::2014-04-11 23:00:25,091::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user) >Thread-395::DEBUG::2014-04-11 23:00:25,092::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`5be6d11d-cbb2-421b-a786-4d1628a24236`::Granted request >Thread-391::DEBUG::2014-04-11 23:00:25,094::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users) >Thread-391::DEBUG::2014-04-11 23:00:25,095::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it. >Thread-391::DEBUG::2014-04-11 23:00:25,096::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records. >Thread-391::DEBUG::2014-04-11 23:00:25,096::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' >Thread-391::DEBUG::2014-04-11 23:00:25,097::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users) >Thread-391::DEBUG::2014-04-11 23:00:25,097::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it. >Thread-391::DEBUG::2014-04-11 23:00:25,097::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,101::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >Thread-390::DEBUG::2014-04-11 23:00:25,104::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955'] >Thread-390::DEBUG::2014-04-11 23:00:25,105::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-390::DEBUG::2014-04-11 23:00:25,106::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-390::DEBUG::2014-04-11 23:00:25,106::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-390::DEBUG::2014-04-11 23:00:25,106::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-390::DEBUG::2014-04-11 23:00:25,107::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-390::DEBUG::2014-04-11 23:00:25,107::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-390::DEBUG::2014-04-11 23:00:25,107::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-390::DEBUG::2014-04-11 23:00:25,108::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-392::DEBUG::2014-04-11 23:00:25,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40'] >Thread-392::DEBUG::2014-04-11 23:00:25,110::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' >Thread-392::DEBUG::2014-04-11 23:00:25,110::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users) >Thread-392::DEBUG::2014-04-11 23:00:25,111::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it. >Thread-392::DEBUG::2014-04-11 23:00:25,111::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records. >Thread-392::DEBUG::2014-04-11 23:00:25,111::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' >Thread-395::DEBUG::2014-04-11 23:00:25,112::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2'] >Thread-392::DEBUG::2014-04-11 23:00:25,113::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users) >Thread-395::DEBUG::2014-04-11 23:00:25,114::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' >Thread-392::DEBUG::2014-04-11 23:00:25,114::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it. >Thread-392::DEBUG::2014-04-11 23:00:25,115::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records. >Thread-395::DEBUG::2014-04-11 23:00:25,115::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' (0 active users) >Thread-395::DEBUG::2014-04-11 23:00:25,116::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it. >Thread-395::DEBUG::2014-04-11 23:00:25,116::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393', Clearing records. >Thread-395::DEBUG::2014-04-11 23:00:25,117::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' >Thread-395::DEBUG::2014-04-11 23:00:25,117::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' (0 active users) >Thread-395::DEBUG::2014-04-11 23:00:25,117::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it. >Thread-395::DEBUG::2014-04-11 23:00:25,118::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393', Clearing records. >Thread-393::DEBUG::2014-04-11 23:00:25,320::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-393::WARNING::2014-04-11 23:00:25,322::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-394::DEBUG::2014-04-11 23:00:25,358::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-393::ERROR::2014-04-11 23:00:25,359::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,363::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,367::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-394::WARNING::2014-04-11 23:00:25,371::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,373::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,399::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-394::ERROR::2014-04-11 23:00:25,429::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,679::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >0a360c1f-47b6-41f6-a72e-e92a32263e98::WARNING::2014-04-11 23:00:25,681::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,681::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,714::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,714::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,716::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:25,717::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,717::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,718::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 23:00:25,866::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:25,868::task::595::TaskManager.Task::(_updateState) Task=`31adae1f-0705-498a-805e-436ecda55a34`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:25,868::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='0a360c1f-47b6-41f6-a72e-e92a32263e98', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:25,868::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 0a360c1f-47b6-41f6-a72e-e92a32263e98 >Thread-14::DEBUG::2014-04-11 23:00:25,869::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'} >Thread-14::INFO::2014-04-11 23:00:25,869::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}} >Thread-14::DEBUG::2014-04-11 23:00:25,870::task::1185::TaskManager.Task::(prepare) Task=`31adae1f-0705-498a-805e-436ecda55a34`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}} >Thread-14::DEBUG::2014-04-11 23:00:25,870::task::595::TaskManager.Task::(_updateState) Task=`31adae1f-0705-498a-805e-436ecda55a34`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:25,870::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:25,871::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:25,871::task::990::TaskManager.Task::(_decref) Task=`31adae1f-0705-498a-805e-436ecda55a34`::ref 0 aborting False >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:25,986::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >0a360c1f-47b6-41f6-a72e-e92a32263e98::WARNING::2014-04-11 23:00:26,000::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,000::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:26,035::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:26,036::sp::329::Storage.StoragePool::(startSpm) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm > self._updateDomainsRole() > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole > domain = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:26,037::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,037::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,038::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`545fbe48-1e92-47b2-b3e9-4090fed5bb56`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,038::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,039::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,039::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`545fbe48-1e92-47b2-b3e9-4090fed5bb56`::Granted request >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,040::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`c55ebd22-d893-4f3f-8b7b-90a5139d7ac3`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,040::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' for lock type 'exclusive' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,041::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free. Now locking as 'exclusive' (1 active user) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,041::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`c55ebd22-d893-4f3f-8b7b-90a5139d7ac3`::Granted request >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,042::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,042::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' (0 active users) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,043::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free, finding out if anyone is waiting for it. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,043::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb', Clearing records. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,044::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`192c4567-9ad1-4b2f-bdac-b86dc3b60438`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,044::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' for lock type 'exclusive' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,044::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free. Now locking as 'exclusive' (1 active user) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,045::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`192c4567-9ad1-4b2f-bdac-b86dc3b60438`::Granted request >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,045::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,046::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' (0 active users) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,046::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free, finding out if anyone is waiting for it. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,046::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48', Clearing records. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,047::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,047::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,048::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,048::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,049::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,050::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,050::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,059::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >0a360c1f-47b6-41f6-a72e-e92a32263e98::INFO::2014-04-11 23:00:26,059::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,062::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released >0a360c1f-47b6-41f6-a72e-e92a32263e98::ERROR::2014-04-11 23:00:26,063::task::866::TaskManager.Task::(_setError) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/storage/task.py", line 334, in run > return self.cmd(*self.argslist, **self.argsdict) > File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm > self._updateDomainsRole() > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole > domain = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,064::task::885::TaskManager.Task::(_run) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Task._run: 0a360c1f-47b6-41f6-a72e-e92a32263e98 () {} failed - stopping task >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,064::task::1211::TaskManager.Task::(stop) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::stopping in state running (force False) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,065::task::990::TaskManager.Task::(_decref) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::ref 1 aborting True >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,065::task::916::TaskManager.Task::(_runJobs) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,065::task::990::TaskManager.Task::(_decref) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::ref 0 aborting True >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,066::task::925::TaskManager.Task::(_doAbort) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Task._doAbort: force False >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,066::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,066::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state running -> state aborting >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,067::task::550::TaskManager.Task::(__state_aborting) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::_aborting: recover policy auto >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,067::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state aborting -> state racquiring >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,067::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state racquiring -> state recovering >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,068::task::798::TaskManager.Task::(_recover) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::_recover >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,068::task::805::TaskManager.Task::(_recover) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::running recovery None >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,068::task::786::TaskManager.Task::(_recoverDone) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::Recover Done: state recovering >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,069::task::595::TaskManager.Task::(_updateState) Task=`0a360c1f-47b6-41f6-a72e-e92a32263e98`::moving from state recovering -> state recovered >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,069::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>} >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,069::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,070::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,070::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,070::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,071::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >0a360c1f-47b6-41f6-a72e-e92a32263e98::DEBUG::2014-04-11 23:00:26,071::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-14::DEBUG::2014-04-11 23:00:26,876::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:26,877::task::595::TaskManager.Task::(_updateState) Task=`50293260-22a4-4c28-8b44-487ae2f5fcca`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:26,878::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='0a360c1f-47b6-41f6-a72e-e92a32263e98', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:26,878::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 0a360c1f-47b6-41f6-a72e-e92a32263e98 >Thread-14::DEBUG::2014-04-11 23:00:26,878::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'} >Thread-14::INFO::2014-04-11 23:00:26,879::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}} >Thread-14::DEBUG::2014-04-11 23:00:26,879::task::1185::TaskManager.Task::(prepare) Task=`50293260-22a4-4c28-8b44-487ae2f5fcca`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '0a360c1f-47b6-41f6-a72e-e92a32263e98'}} >Thread-14::DEBUG::2014-04-11 23:00:26,879::task::595::TaskManager.Task::(_updateState) Task=`50293260-22a4-4c28-8b44-487ae2f5fcca`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:26,880::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:26,880::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:26,880::task::990::TaskManager.Task::(_decref) Task=`50293260-22a4-4c28-8b44-487ae2f5fcca`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:26,894::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:26,899::task::595::TaskManager.Task::(_updateState) Task=`d20f7218-ba2a-4a82-99bb-0c17cfcf6ca8`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:26,902::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:26,920::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:26,923::task::1185::TaskManager.Task::(prepare) Task=`d20f7218-ba2a-4a82-99bb-0c17cfcf6ca8`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:26,925::task::595::TaskManager.Task::(_updateState) Task=`d20f7218-ba2a-4a82-99bb-0c17cfcf6ca8`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:26,928::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:26,931::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:26,934::task::990::TaskManager.Task::(_decref) Task=`d20f7218-ba2a-4a82-99bb-0c17cfcf6ca8`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:27,042::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:27,043::task::595::TaskManager.Task::(_updateState) Task=`1fec0d62-3d9e-4ef4-a9ca-92f9ac4a9e5a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:27,043::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='0a360c1f-47b6-41f6-a72e-e92a32263e98', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:27,044::taskManager::161::TaskManager::(clearTask) Entry. taskID: 0a360c1f-47b6-41f6-a72e-e92a32263e98 >Thread-14::DEBUG::2014-04-11 23:00:27,044::taskManager::166::TaskManager::(clearTask) Return. >Thread-14::INFO::2014-04-11 23:00:27,044::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-14::DEBUG::2014-04-11 23:00:27,045::task::1185::TaskManager.Task::(prepare) Task=`1fec0d62-3d9e-4ef4-a9ca-92f9ac4a9e5a`::finished: None >Thread-14::DEBUG::2014-04-11 23:00:27,045::task::595::TaskManager.Task::(_updateState) Task=`1fec0d62-3d9e-4ef4-a9ca-92f9ac4a9e5a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:27,045::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:27,046::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:27,046::task::990::TaskManager.Task::(_decref) Task=`1fec0d62-3d9e-4ef4-a9ca-92f9ac4a9e5a`::ref 0 aborting False >Thread-33::ERROR::2014-04-11 23:00:29,125::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 23:00:29,125::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 23:00:29,126::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33::DEBUG::2014-04-11 23:00:29,127::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-39::ERROR::2014-04-11 23:00:29,186::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 23:00:29,187::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 23:00:29,188::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-39::DEBUG::2014-04-11 23:00:29,190::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-33::DEBUG::2014-04-11 23:00:29,397::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 23:00:29,399::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::ERROR::2014-04-11 23:00:29,431::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 23:00:29,432::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-39::DEBUG::2014-04-11 23:00:29,465::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 23:00:29,467::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::DEBUG::2014-04-11 23:00:29,467::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39::ERROR::2014-04-11 23:00:29,500::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 23:00:29,500::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-43::DEBUG::2014-04-11 23:00:32,686::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 23:00:32,700::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000495644 s, 1.5 MB/s\n'; <rc> = 0 >Thread-24::DEBUG::2014-04-11 23:00:33,509::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 23:00:33,522::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000393856 s, 937 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 23:00:33,678::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 23:00:33,691::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000395372 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 23:00:34,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 23:00:34,098::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000358786 s, 2.1 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 23:00:34,130::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 23:00:34,142::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000328121 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 23:00:37,206::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:37,245::task::595::TaskManager.Task::(_updateState) Task=`1ce9ac53-3258-432b-a837-283dd83d34dd`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:37,245::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:37,261::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:37,261::task::1185::TaskManager.Task::(prepare) Task=`1ce9ac53-3258-432b-a837-283dd83d34dd`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:37,262::task::595::TaskManager.Task::(_updateState) Task=`1ce9ac53-3258-432b-a837-283dd83d34dd`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:37,262::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:37,262::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:37,263::task::990::TaskManager.Task::(_decref) Task=`1ce9ac53-3258-432b-a837-283dd83d34dd`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:37,267::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:37,268::task::595::TaskManager.Task::(_updateState) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:37,268::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-14::ERROR::2014-04-11 23:00:37,268::task::866::TaskManager.Task::(_setError) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2115, in getAllTasksStatuses > allTasksStatus = sp.getAllTasksStatuses() > File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper > raise SecureError("Secured object is not in safe state") >SecureError: Secured object is not in safe state >Thread-14::DEBUG::2014-04-11 23:00:37,269::task::885::TaskManager.Task::(_run) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::Task._run: 05cf4d7b-f676-47ac-8313-8941fca0ab57 () {} failed - stopping task >Thread-14::DEBUG::2014-04-11 23:00:37,269::task::1211::TaskManager.Task::(stop) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 23:00:37,270::task::990::TaskManager.Task::(_decref) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::ref 1 aborting True >Thread-14::INFO::2014-04-11 23:00:37,270::task::1168::TaskManager.Task::(prepare) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100 >Thread-14::DEBUG::2014-04-11 23:00:37,270::task::1173::TaskManager.Task::(prepare) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::Prepare: aborted: Secured object is not in safe state >Thread-14::DEBUG::2014-04-11 23:00:37,271::task::990::TaskManager.Task::(_decref) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 23:00:37,271::task::925::TaskManager.Task::(_doAbort) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 23:00:37,271::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:37,271::task::595::TaskManager.Task::(_updateState) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 23:00:37,272::task::550::TaskManager.Task::(__state_aborting) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 23:00:37,272::task::595::TaskManager.Task::(_updateState) Task=`05cf4d7b-f676-47ac-8313-8941fca0ab57`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 23:00:37,272::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:37,272::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 23:00:37,273::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SecureError: Secured object is not in safe state >Thread-14::DEBUG::2014-04-11 23:00:37,392::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:37,393::task::595::TaskManager.Task::(_updateState) Task=`f85ca92a-8f57-4e28-854d-282e00bb8308`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:37,393::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:37,406::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:37,407::task::1185::TaskManager.Task::(prepare) Task=`f85ca92a-8f57-4e28-854d-282e00bb8308`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:37,407::task::595::TaskManager.Task::(_updateState) Task=`f85ca92a-8f57-4e28-854d-282e00bb8308`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:37,407::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:37,407::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:37,408::task::990::TaskManager.Task::(_decref) Task=`f85ca92a-8f57-4e28-854d-282e00bb8308`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:37,456::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:37,457::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:37,457::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None) >Thread-14::DEBUG::2014-04-11 23:00:37,458::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`1e904999-00f0-4c11-9fbc-28e0a09e92d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart' >Thread-14::DEBUG::2014-04-11 23:00:37,459::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 23:00:37,459::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 23:00:37,459::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`1e904999-00f0-4c11-9fbc-28e0a09e92d2`::Granted request >Thread-14::DEBUG::2014-04-11 23:00:37,460::task::827::TaskManager.Task::(resourceAcquired) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive) >Thread-14::DEBUG::2014-04-11 23:00:37,460::task::990::TaskManager.Task::(_decref) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::ref 1 aborting False >Thread-14::INFO::2014-04-11 23:00:37,461::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-14::DEBUG::2014-04-11 23:00:37,461::task::1180::TaskManager.Task::(prepare) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Prepare: 1 jobs exist, move to acquiring >Thread-14::DEBUG::2014-04-11 23:00:37,461::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state preparing -> state acquiring >Thread-14::DEBUG::2014-04-11 23:00:37,462::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state acquiring -> state queued >Thread-14::DEBUG::2014-04-11 23:00:37,462::taskManager::50::TaskManager::(_queueTask) queuing task: 9ceac723-a09b-426b-9729-c3d1775660a9 >Thread-14::DEBUG::2014-04-11 23:00:37,462::taskManager::56::TaskManager::(_queueTask) task queued: 9ceac723-a09b-426b-9729-c3d1775660a9 >Thread-10::DEBUG::2014-04-11 23:00:37,463::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-14::DEBUG::2014-04-11 23:00:37,463::task::1182::TaskManager.Task::(prepare) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::returning >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,463::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 9ceac723-a09b-426b-9729-c3d1775660a9 running: <bound method Task.commit of <storage.task.Task instance at 0x2e38878>> with: None >Thread-14::DEBUG::2014-04-11 23:00:37,464::task::990::TaskManager.Task::(_decref) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::ref 0 aborting False >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,465::task::1194::TaskManager.Task::(commit) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::committing task: 9ceac723-a09b-426b-9729-c3d1775660a9 >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,466::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state queued -> state running >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,466::task::905::TaskManager.Task::(_runJobs) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,467::task::331::TaskManager.Task::(run) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,468::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,468::misc::764::SamplingMethod::(__call__) Got in to sampling method >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,486::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,486::misc::772::SamplingMethod::(__call__) Returning last result >9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,499::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,500::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,500::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,543::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,543::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,559::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,559::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,560::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,560::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,583::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,584::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`74f47895-90e6-4be0-9bb1-a37fe6924576`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,584::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,585::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,585::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`74f47895-90e6-4be0-9bb1-a37fe6924576`::Granted request >9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,586::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a` >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,586::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`4df9b8aa-031b-4b60-bc6b-186a827cda7d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,587::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,587::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,587::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`4df9b8aa-031b-4b60-bc6b-186a827cda7d`::Granted request >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,602::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,604::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,620::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,622::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,622::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered >9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:37,623::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,623::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,623::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,624::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,624::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,624::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,640::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,640::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,641::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >Thread-438::DEBUG::2014-04-11 23:00:37,641::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-438::DEBUG::2014-04-11 23:00:37,643::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`552f1acb-2df2-4ca5-82dc-4eeb71313107`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-439::DEBUG::2014-04-11 23:00:37,643::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-439::DEBUG::2014-04-11 23:00:37,645::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`92c2227e-8e51-4308-8d5d-38a6a1bfed5e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-438::DEBUG::2014-04-11 23:00:37,645::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive' >Thread-440::DEBUG::2014-04-11 23:00:37,645::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-439::DEBUG::2014-04-11 23:00:37,647::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive' >Thread-438::DEBUG::2014-04-11 23:00:37,647::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user) >Thread-441::ERROR::2014-04-11 23:00:37,647::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-442::ERROR::2014-04-11 23:00:37,649::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-440::DEBUG::2014-04-11 23:00:37,651::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`01d8ffab-8ab4-4066-a37b-6008760bdd06`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-438::DEBUG::2014-04-11 23:00:37,652::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`552f1acb-2df2-4ca5-82dc-4eeb71313107`::Granted request >Thread-443::DEBUG::2014-04-11 23:00:37,652::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 21619c8e-99ea-4813-be02-d708971e5393 >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,652::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-441::ERROR::2014-04-11 23:00:37,653::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-442::ERROR::2014-04-11 23:00:37,653::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-440::DEBUG::2014-04-11 23:00:37,653::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive' >Thread-443::DEBUG::2014-04-11 23:00:37,655::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`31a554ec-565a-4d91-bef3-01e993fc2155`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-439::DEBUG::2014-04-11 23:00:37,655::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user) >Thread-441::DEBUG::2014-04-11 23:00:37,656::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-438::DEBUG::2014-04-11 23:00:37,657::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`e26de70a-5033-4dac-a9be-7b4f73b8ee3f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-443::DEBUG::2014-04-11 23:00:37,658::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive' >Thread-439::DEBUG::2014-04-11 23:00:37,658::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`92c2227e-8e51-4308-8d5d-38a6a1bfed5e`::Granted request >Thread-442::DEBUG::2014-04-11 23:00:37,659::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-441::DEBUG::2014-04-11 23:00:37,660::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-438::DEBUG::2014-04-11 23:00:37,660::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,661::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-442::DEBUG::2014-04-11 23:00:37,662::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-439::DEBUG::2014-04-11 23:00:37,663::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`dcf38eae-4367-4806-95c8-6b387fef2fb1`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,676::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-439::DEBUG::2014-04-11 23:00:37,687::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,688::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-440::DEBUG::2014-04-11 23:00:37,690::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user) >Thread-440::DEBUG::2014-04-11 23:00:37,691::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`01d8ffab-8ab4-4066-a37b-6008760bdd06`::Granted request >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,692::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms >Thread-443::DEBUG::2014-04-11 23:00:37,694::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user) >Thread-440::DEBUG::2014-04-11 23:00:37,695::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`e75021d1-c0a3-4edd-a594-da6ed73d96ac`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-443::DEBUG::2014-04-11 23:00:37,695::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`31a554ec-565a-4d91-bef3-01e993fc2155`::Granted request >Thread-440::DEBUG::2014-04-11 23:00:37,696::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,696::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks >Thread-438::DEBUG::2014-04-11 23:00:37,697::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user) >Thread-438::DEBUG::2014-04-11 23:00:37,700::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`e26de70a-5033-4dac-a9be-7b4f73b8ee3f`::Granted request >Thread-443::DEBUG::2014-04-11 23:00:37,701::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`cccb83fe-4a9f-4f71-b5a6-78c659ad1668`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-443::DEBUG::2014-04-11 23:00:37,702::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive' >Thread-439::DEBUG::2014-04-11 23:00:37,703::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user) >Thread-439::DEBUG::2014-04-11 23:00:37,704::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`dcf38eae-4367-4806-95c8-6b387fef2fb1`::Granted request >Thread-439::DEBUG::2014-04-11 23:00:37,705::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` is not a data domain it is an 2 domain, not upgrading >Thread-440::DEBUG::2014-04-11 23:00:37,705::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user) >Thread-439::DEBUG::2014-04-11 23:00:37,706::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' >Thread-440::DEBUG::2014-04-11 23:00:37,706::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`e75021d1-c0a3-4edd-a594-da6ed73d96ac`::Granted request >Thread-443::DEBUG::2014-04-11 23:00:37,708::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user) >Thread-443::DEBUG::2014-04-11 23:00:37,709::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`cccb83fe-4a9f-4f71-b5a6-78c659ad1668`::Granted request >Thread-439::DEBUG::2014-04-11 23:00:37,711::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users) >Thread-439::DEBUG::2014-04-11 23:00:37,711::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it. >Thread-439::DEBUG::2014-04-11 23:00:37,711::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records. >Thread-439::DEBUG::2014-04-11 23:00:37,712::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' >Thread-439::DEBUG::2014-04-11 23:00:37,712::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users) >Thread-439::DEBUG::2014-04-11 23:00:37,713::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it. >Thread-439::DEBUG::2014-04-11 23:00:37,713::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,717::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >Thread-438::DEBUG::2014-04-11 23:00:37,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955'] >Thread-438::DEBUG::2014-04-11 23:00:37,722::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-438::DEBUG::2014-04-11 23:00:37,723::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-438::DEBUG::2014-04-11 23:00:37,723::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-438::DEBUG::2014-04-11 23:00:37,723::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-438::DEBUG::2014-04-11 23:00:37,724::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-438::DEBUG::2014-04-11 23:00:37,724::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-438::DEBUG::2014-04-11 23:00:37,724::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-438::DEBUG::2014-04-11 23:00:37,725::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-440::DEBUG::2014-04-11 23:00:37,726::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40'] >Thread-440::DEBUG::2014-04-11 23:00:37,727::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' >Thread-440::DEBUG::2014-04-11 23:00:37,727::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users) >Thread-440::DEBUG::2014-04-11 23:00:37,727::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it. >Thread-440::DEBUG::2014-04-11 23:00:37,728::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records. >Thread-440::DEBUG::2014-04-11 23:00:37,728::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' >Thread-440::DEBUG::2014-04-11 23:00:37,728::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users) >Thread-440::DEBUG::2014-04-11 23:00:37,729::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it. >Thread-440::DEBUG::2014-04-11 23:00:37,729::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records. >Thread-443::DEBUG::2014-04-11 23:00:37,730::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2'] >Thread-443::DEBUG::2014-04-11 23:00:37,731::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' >Thread-443::DEBUG::2014-04-11 23:00:37,731::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' (0 active users) >Thread-443::DEBUG::2014-04-11 23:00:37,732::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it. >Thread-443::DEBUG::2014-04-11 23:00:37,732::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393', Clearing records. >Thread-443::DEBUG::2014-04-11 23:00:37,732::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' >Thread-443::DEBUG::2014-04-11 23:00:37,733::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' (0 active users) >Thread-443::DEBUG::2014-04-11 23:00:37,733::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it. >Thread-443::DEBUG::2014-04-11 23:00:37,733::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393', Clearing records. >Thread-441::DEBUG::2014-04-11 23:00:37,942::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-441::WARNING::2014-04-11 23:00:37,944::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-442::DEBUG::2014-04-11 23:00:37,983::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-441::ERROR::2014-04-11 23:00:37,983::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:37,990::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:37,995::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-442::WARNING::2014-04-11 23:00:37,995::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:37,999::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,029::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-442::ERROR::2014-04-11 23:00:38,059::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,309::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >9ceac723-a09b-426b-9729-c3d1775660a9::WARNING::2014-04-11 23:00:38,311::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,311::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,343::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,344::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,346::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,346::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,346::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,347::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 23:00:38,473::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:38,475::task::595::TaskManager.Task::(_updateState) Task=`4895e8fc-c1f1-41c1-ba22-283b27b6e991`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:38,475::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9ceac723-a09b-426b-9729-c3d1775660a9', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:38,476::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9ceac723-a09b-426b-9729-c3d1775660a9 >Thread-14::DEBUG::2014-04-11 23:00:38,476::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'} >Thread-14::INFO::2014-04-11 23:00:38,476::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}} >Thread-14::DEBUG::2014-04-11 23:00:38,477::task::1185::TaskManager.Task::(prepare) Task=`4895e8fc-c1f1-41c1-ba22-283b27b6e991`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}} >Thread-14::DEBUG::2014-04-11 23:00:38,477::task::595::TaskManager.Task::(_updateState) Task=`4895e8fc-c1f1-41c1-ba22-283b27b6e991`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:38,477::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:38,478::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:38,478::task::990::TaskManager.Task::(_decref) Task=`4895e8fc-c1f1-41c1-ba22-283b27b6e991`::ref 0 aborting False >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,615::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >9ceac723-a09b-426b-9729-c3d1775660a9::WARNING::2014-04-11 23:00:38,617::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,617::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,655::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,658::sp::329::Storage.StoragePool::(startSpm) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm > self._updateDomainsRole() > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole > domain = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 23:00:38,661::task::595::TaskManager.Task::(_updateState) Task=`e9064fc3-7f9d-4a81-8cf0-f9a7677ab3b6`::moving from state init -> state preparing >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,662::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,664::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process >Thread-14::INFO::2014-04-11 23:00:38,664::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,668::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`8629862c-6c9e-438d-8455-c27a6a9d082a`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,670::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::INFO::2014-04-11 23:00:38,670::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000395372', 'lastCheck': '5.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393856', 'lastCheck': '5.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000358786', 'lastCheck': '4.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000495644', 'lastCheck': '6.0', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000328121', 'lastCheck': '4.5', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:38,672::task::1185::TaskManager.Task::(prepare) Task=`e9064fc3-7f9d-4a81-8cf0-f9a7677ab3b6`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000395372', 'lastCheck': '5.0', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393856', 'lastCheck': '5.1', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000358786', 'lastCheck': '4.6', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000495644', 'lastCheck': '6.0', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000328121', 'lastCheck': '4.5', 'valid': True}} >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,673::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 23:00:38,674::task::595::TaskManager.Task::(_updateState) Task=`e9064fc3-7f9d-4a81-8cf0-f9a7677ab3b6`::moving from state preparing -> state finished >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,675::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`8629862c-6c9e-438d-8455-c27a6a9d082a`::Granted request >Thread-14::DEBUG::2014-04-11 23:00:38,676::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:38,678::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,681::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`e717d49d-f713-4fe4-a096-b03ed289c70e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' >Thread-14::DEBUG::2014-04-11 23:00:38,681::task::990::TaskManager.Task::(_decref) Task=`e9064fc3-7f9d-4a81-8cf0-f9a7677ab3b6`::ref 0 aborting False >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,682::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' for lock type 'exclusive' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,685::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free. Now locking as 'exclusive' (1 active user) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,688::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`e717d49d-f713-4fe4-a096-b03ed289c70e`::Granted request >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,693::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,696::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' (0 active users) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,698::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free, finding out if anyone is waiting for it. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,700::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb', Clearing records. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,705::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`b8633a53-53cd-4982-bb93-c2efd85bb46f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,707::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' for lock type 'exclusive' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,710::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free. Now locking as 'exclusive' (1 active user) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,712::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`b8633a53-53cd-4982-bb93-c2efd85bb46f`::Granted request >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,716::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,719::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' (0 active users) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,721::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free, finding out if anyone is waiting for it. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,723::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48', Clearing records. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,728::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,730::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,731::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,732::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,733::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,733::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,734::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,743::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >9ceac723-a09b-426b-9729-c3d1775660a9::INFO::2014-04-11 23:00:38,743::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,746::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released >9ceac723-a09b-426b-9729-c3d1775660a9::ERROR::2014-04-11 23:00:38,746::task::866::TaskManager.Task::(_setError) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/storage/task.py", line 334, in run > return self.cmd(*self.argslist, **self.argsdict) > File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm > self._updateDomainsRole() > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole > domain = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,747::task::885::TaskManager.Task::(_run) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Task._run: 9ceac723-a09b-426b-9729-c3d1775660a9 () {} failed - stopping task >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,748::task::1211::TaskManager.Task::(stop) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::stopping in state running (force False) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,749::task::990::TaskManager.Task::(_decref) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::ref 1 aborting True >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,751::task::916::TaskManager.Task::(_runJobs) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,753::task::990::TaskManager.Task::(_decref) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::ref 0 aborting True >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,755::task::925::TaskManager.Task::(_doAbort) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Task._doAbort: force False >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,756::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,756::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state running -> state aborting >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,756::task::550::TaskManager.Task::(__state_aborting) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::_aborting: recover policy auto >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,757::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state aborting -> state racquiring >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,757::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state racquiring -> state recovering >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,757::task::798::TaskManager.Task::(_recover) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::_recover >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,758::task::805::TaskManager.Task::(_recover) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::running recovery None >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,758::task::786::TaskManager.Task::(_recoverDone) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::Recover Done: state recovering >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,758::task::595::TaskManager.Task::(_updateState) Task=`9ceac723-a09b-426b-9729-c3d1775660a9`::moving from state recovering -> state recovered >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,758::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>} >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,759::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,759::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,760::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,760::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,760::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:38,761::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-33::ERROR::2014-04-11 23:00:39,433::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 23:00:39,434::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 23:00:39,434::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33::DEBUG::2014-04-11 23:00:39,436::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 23:00:39,485::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:39,486::task::595::TaskManager.Task::(_updateState) Task=`9c0a6294-2b54-4799-8114-f6619ad69b58`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:39,486::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='9ceac723-a09b-426b-9729-c3d1775660a9', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:39,487::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 9ceac723-a09b-426b-9729-c3d1775660a9 >Thread-14::DEBUG::2014-04-11 23:00:39,487::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'} >Thread-14::INFO::2014-04-11 23:00:39,488::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}} >Thread-14::DEBUG::2014-04-11 23:00:39,488::task::1185::TaskManager.Task::(prepare) Task=`9c0a6294-2b54-4799-8114-f6619ad69b58`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '9ceac723-a09b-426b-9729-c3d1775660a9'}} >Thread-14::DEBUG::2014-04-11 23:00:39,488::task::595::TaskManager.Task::(_updateState) Task=`9c0a6294-2b54-4799-8114-f6619ad69b58`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:39,489::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:39,489::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:39,489::task::990::TaskManager.Task::(_decref) Task=`9c0a6294-2b54-4799-8114-f6619ad69b58`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:39,500::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:39,501::task::595::TaskManager.Task::(_updateState) Task=`32f221a2-8e14-4eaf-b713-6c0afdea02d2`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:39,502::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-39::ERROR::2014-04-11 23:00:39,502::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 23:00:39,505::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 23:00:39,505::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-39::DEBUG::2014-04-11 23:00:39,507::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-14::INFO::2014-04-11 23:00:39,530::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:39,531::task::1185::TaskManager.Task::(prepare) Task=`32f221a2-8e14-4eaf-b713-6c0afdea02d2`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:39,532::task::595::TaskManager.Task::(_updateState) Task=`32f221a2-8e14-4eaf-b713-6c0afdea02d2`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:39,533::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:39,533::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:39,534::task::990::TaskManager.Task::(_decref) Task=`32f221a2-8e14-4eaf-b713-6c0afdea02d2`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:39,616::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-14::DEBUG::2014-04-11 23:00:39,618::task::595::TaskManager.Task::(_updateState) Task=`18e6c9c0-1975-4001-b38a-0f81fd194d2c`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:39,618::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='9ceac723-a09b-426b-9729-c3d1775660a9', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:39,619::taskManager::161::TaskManager::(clearTask) Entry. taskID: 9ceac723-a09b-426b-9729-c3d1775660a9 >Thread-14::DEBUG::2014-04-11 23:00:39,619::taskManager::166::TaskManager::(clearTask) Return. >Thread-14::INFO::2014-04-11 23:00:39,620::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-14::DEBUG::2014-04-11 23:00:39,621::task::1185::TaskManager.Task::(prepare) Task=`18e6c9c0-1975-4001-b38a-0f81fd194d2c`::finished: None >Thread-14::DEBUG::2014-04-11 23:00:39,621::task::595::TaskManager.Task::(_updateState) Task=`18e6c9c0-1975-4001-b38a-0f81fd194d2c`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:39,622::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:39,622::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:39,623::task::990::TaskManager.Task::(_decref) Task=`18e6c9c0-1975-4001-b38a-0f81fd194d2c`::ref 0 aborting False >Thread-33::DEBUG::2014-04-11 23:00:39,708::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 23:00:39,710::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-33::ERROR::2014-04-11 23:00:39,743::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-33::ERROR::2014-04-11 23:00:39,744::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-39::DEBUG::2014-04-11 23:00:39,790::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 23:00:39,801::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::DEBUG::2014-04-11 23:00:39,802::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39::ERROR::2014-04-11 23:00:39,842::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 23:00:39,842::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-43::DEBUG::2014-04-11 23:00:42,725::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 23:00:42,739::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000453198 s, 1.7 MB/s\n'; <rc> = 0 >Thread-24::DEBUG::2014-04-11 23:00:43,550::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 23:00:43,564::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000362865 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 23:00:43,720::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 23:00:43,734::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000599656 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-04-11 23:00:44,131::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 23:00:44,143::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000564156 s, 1.3 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 23:00:44,174::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 23:00:44,185::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000302579 s, 2.5 MB/s\n'; <rc> = 0 >Thread-33::ERROR::2014-04-11 23:00:49,745::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::ERROR::2014-04-11 23:00:49,746::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 23:00:49,747::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33::DEBUG::2014-04-11 23:00:49,748::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 23:00:49,799::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61] >Thread-14::DEBUG::2014-04-11 23:00:49,800::task::595::TaskManager.Task::(_updateState) Task=`f3870501-e6df-4d6f-a67a-8b12780d422a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:49,801::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:49,816::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:49,817::task::1185::TaskManager.Task::(prepare) Task=`f3870501-e6df-4d6f-a67a-8b12780d422a`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:49,817::task::595::TaskManager.Task::(_updateState) Task=`f3870501-e6df-4d6f-a67a-8b12780d422a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:49,817::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:49,818::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:49,818::task::990::TaskManager.Task::(_decref) Task=`f3870501-e6df-4d6f-a67a-8b12780d422a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:49,823::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61] >Thread-14::DEBUG::2014-04-11 23:00:49,823::task::595::TaskManager.Task::(_updateState) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:49,824::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-14::ERROR::2014-04-11 23:00:49,824::task::866::TaskManager.Task::(_setError) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2115, in getAllTasksStatuses > allTasksStatus = sp.getAllTasksStatuses() > File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper > raise SecureError("Secured object is not in safe state") >SecureError: Secured object is not in safe state >Thread-14::DEBUG::2014-04-11 23:00:49,825::task::885::TaskManager.Task::(_run) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::Task._run: e987ef15-ed21-40b3-9b34-c2489cd0c106 () {} failed - stopping task >Thread-14::DEBUG::2014-04-11 23:00:49,825::task::1211::TaskManager.Task::(stop) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::stopping in state preparing (force False) >Thread-14::DEBUG::2014-04-11 23:00:49,826::task::990::TaskManager.Task::(_decref) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::ref 1 aborting True >Thread-14::INFO::2014-04-11 23:00:49,826::task::1168::TaskManager.Task::(prepare) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100 >Thread-14::DEBUG::2014-04-11 23:00:49,826::task::1173::TaskManager.Task::(prepare) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::Prepare: aborted: Secured object is not in safe state >Thread-14::DEBUG::2014-04-11 23:00:49,827::task::990::TaskManager.Task::(_decref) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::ref 0 aborting True >Thread-14::DEBUG::2014-04-11 23:00:49,827::task::925::TaskManager.Task::(_doAbort) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::Task._doAbort: force False >Thread-14::DEBUG::2014-04-11 23:00:49,827::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:49,828::task::595::TaskManager.Task::(_updateState) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::moving from state preparing -> state aborting >Thread-14::DEBUG::2014-04-11 23:00:49,828::task::550::TaskManager.Task::(__state_aborting) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::_aborting: recover policy none >Thread-14::DEBUG::2014-04-11 23:00:49,828::task::595::TaskManager.Task::(_updateState) Task=`e987ef15-ed21-40b3-9b34-c2489cd0c106`::moving from state aborting -> state failed >Thread-14::DEBUG::2014-04-11 23:00:49,829::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:49,829::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::ERROR::2014-04-11 23:00:49,829::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SecureError: Secured object is not in safe state >Thread-39::ERROR::2014-04-11 23:00:49,843::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 23:00:49,844::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 23:00:49,844::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-39::DEBUG::2014-04-11 23:00:49,845::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-14::DEBUG::2014-04-11 23:00:49,953::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61] >Thread-14::DEBUG::2014-04-11 23:00:49,955::task::595::TaskManager.Task::(_updateState) Task=`164fe45f-522d-450e-8b83-9968c6c4241d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:49,956::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:49,970::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:49,970::task::1185::TaskManager.Task::(prepare) Task=`164fe45f-522d-450e-8b83-9968c6c4241d`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:49,971::task::595::TaskManager.Task::(_updateState) Task=`164fe45f-522d-450e-8b83-9968c6c4241d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:49,971::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:49,972::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:49,973::task::990::TaskManager.Task::(_decref) Task=`164fe45f-522d-450e-8b83-9968c6c4241d`::ref 0 aborting False >Thread-33::DEBUG::2014-04-11 23:00:50,021::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-33::WARNING::2014-04-11 23:00:50,024::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-14::DEBUG::2014-04-11 23:00:50,050::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61] >Thread-14::DEBUG::2014-04-11 23:00:50,062::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state init -> state preparing >Thread-33::ERROR::2014-04-11 23:00:50,065::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::INFO::2014-04-11 23:00:50,068::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None) >Thread-33::ERROR::2014-04-11 23:00:50,070::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 3406665e-4adc-4fd4-aa1e-037547b29adb monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >Thread-14::DEBUG::2014-04-11 23:00:50,076::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`59d35f2b-7bbc-490d-ad3f-b3a4ba831fe3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '578' at 'spmStart' >Thread-14::DEBUG::2014-04-11 23:00:50,077::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 23:00:50,077::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 23:00:50,078::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`59d35f2b-7bbc-490d-ad3f-b3a4ba831fe3`::Granted request >Thread-14::DEBUG::2014-04-11 23:00:50,078::task::827::TaskManager.Task::(resourceAcquired) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive) >Thread-14::DEBUG::2014-04-11 23:00:50,079::task::990::TaskManager.Task::(_decref) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::ref 1 aborting False >Thread-14::INFO::2014-04-11 23:00:50,079::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-14::DEBUG::2014-04-11 23:00:50,080::task::1180::TaskManager.Task::(prepare) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Prepare: 1 jobs exist, move to acquiring >Thread-14::DEBUG::2014-04-11 23:00:50,080::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state preparing -> state acquiring >Thread-14::DEBUG::2014-04-11 23:00:50,080::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state acquiring -> state queued >Thread-14::DEBUG::2014-04-11 23:00:50,080::taskManager::50::TaskManager::(_queueTask) queuing task: 8be642d5-95c4-4237-9639-a2844d95ee69 >Thread-14::DEBUG::2014-04-11 23:00:50,081::taskManager::56::TaskManager::(_queueTask) task queued: 8be642d5-95c4-4237-9639-a2844d95ee69 >9ceac723-a09b-426b-9729-c3d1775660a9::DEBUG::2014-04-11 23:00:50,081::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-14::DEBUG::2014-04-11 23:00:50,083::task::1182::TaskManager.Task::(prepare) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::returning >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,084::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 8be642d5-95c4-4237-9639-a2844d95ee69 running: <bound method Task.commit of <storage.task.Task instance at 0x7fc3f8169830>> with: None >Thread-14::DEBUG::2014-04-11 23:00:50,085::task::990::TaskManager.Task::(_decref) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::ref 0 aborting False >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,086::task::1194::TaskManager.Task::(commit) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::committing task: 8be642d5-95c4-4237-9639-a2844d95ee69 >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,088::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state queued -> state running >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,090::task::905::TaskManager.Task::(_runJobs) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,092::task::331::TaskManager.Task::(run) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fc414050dd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,092::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,093::misc::764::SamplingMethod::(__call__) Got in to sampling method >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,111::misc::772::SamplingMethod::(__call__) Returning last result >Thread-39::DEBUG::2014-04-11 23:00:50,123::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,126::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,129::clusterlock::202::SANLock::(acquireHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >Thread-39::WARNING::2014-04-11 23:00:50,131::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,134::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-39::DEBUG::2014-04-11 23:00:50,136::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39::ERROR::2014-04-11 23:00:50,170::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 23:00:50,171::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,187::clusterlock::261::SANLock::(acquire) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully acquired (id: 1) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,187::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,203::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,204::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,205::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,205::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,221::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,222::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`2977357f-55fd-4820-895a-a161723b8c1e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,223::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,223::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,223::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`2977357f-55fd-4820-895a-a161723b8c1e`::Granted request >8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,224::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a033c2ac-0d01-490c-9552-99ca53d6a64a` >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,225::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`7374d821-417d-42e1-9905-7af8bfa8600a`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,225::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' for lock type 'exclusive' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,226::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free. Now locking as 'exclusive' (1 active user) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,226::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a`ReqID=`7374d821-417d-42e1-9905-7af8bfa8600a`::Granted request >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,266::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,267::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,284::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,287::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,287::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_imageNS already registered >8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:50,287::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a033c2ac-0d01-490c-9552-99ca53d6a64a_volumeNS already registered >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,288::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,288::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' (0 active users) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,289::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a' is free, finding out if anyone is waiting for it. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,289::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a033c2ac-0d01-490c-9552-99ca53d6a64a', Clearing records. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,289::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,305::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,306::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,306::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >Thread-498::DEBUG::2014-04-11 23:00:50,308::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-499::DEBUG::2014-04-11 23:00:50,310::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-500::DEBUG::2014-04-11 23:00:50,314::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-501::ERROR::2014-04-11 23:00:50,319::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-502::ERROR::2014-04-11 23:00:50,324::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-501::ERROR::2014-04-11 23:00:50,328::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-503::DEBUG::2014-04-11 23:00:50,330::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 21619c8e-99ea-4813-be02-d708971e5393 >Thread-498::DEBUG::2014-04-11 23:00:50,331::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`60963b15-5eb8-487c-b63a-903638f2e361`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-502::ERROR::2014-04-11 23:00:50,332::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,334::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-499::DEBUG::2014-04-11 23:00:50,335::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`c8663709-ef61-4309-9fbf-fae6c5053f49`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-500::DEBUG::2014-04-11 23:00:50,339::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`4274a941-8ceb-4b3f-ab73-1a196c9b1a57`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >Thread-498::DEBUG::2014-04-11 23:00:50,342::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive' >Thread-501::DEBUG::2014-04-11 23:00:50,342::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-499::DEBUG::2014-04-11 23:00:50,348::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,349::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-500::DEBUG::2014-04-11 23:00:50,350::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive' >Thread-503::DEBUG::2014-04-11 23:00:50,352::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`5bc81f0a-b542-4901-9a5b-8164e05176cb`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,358::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-502::DEBUG::2014-04-11 23:00:50,361::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-503::DEBUG::2014-04-11 23:00:50,362::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,365::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-498::DEBUG::2014-04-11 23:00:50,372::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user) >Thread-498::DEBUG::2014-04-11 23:00:50,378::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`60963b15-5eb8-487c-b63a-903638f2e361`::Granted request >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,387::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/vms >Thread-499::DEBUG::2014-04-11 23:00:50,388::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user) >Thread-501::DEBUG::2014-04-11 23:00:50,389::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-499::DEBUG::2014-04-11 23:00:50,408::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`c8663709-ef61-4309-9fbf-fae6c5053f49`::Granted request >Thread-502::DEBUG::2014-04-11 23:00:50,418::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-498::DEBUG::2014-04-11 23:00:50,420::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`2db1388a-4df6-4f17-bf71-271537b05f2e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,420::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/master/tasks >Thread-500::DEBUG::2014-04-11 23:00:50,424::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user) >Thread-498::DEBUG::2014-04-11 23:00:50,441::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' for lock type 'exclusive' >Thread-500::DEBUG::2014-04-11 23:00:50,448::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`4274a941-8ceb-4b3f-ab73-1a196c9b1a57`::Granted request >Thread-499::DEBUG::2014-04-11 23:00:50,453::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`139ca213-17f3-4c3a-b8ea-63e7f5bf8511`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-503::DEBUG::2014-04-11 23:00:50,458::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user) >Thread-499::DEBUG::2014-04-11 23:00:50,461::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' for lock type 'exclusive' >Thread-503::DEBUG::2014-04-11 23:00:50,465::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393`ReqID=`5bc81f0a-b542-4901-9a5b-8164e05176cb`::Granted request >Thread-500::DEBUG::2014-04-11 23:00:50,471::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`26a15518-3c6f-4405-987e-44a39be5a581`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-498::DEBUG::2014-04-11 23:00:50,472::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free. Now locking as 'exclusive' (1 active user) >Thread-500::DEBUG::2014-04-11 23:00:50,474::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' for lock type 'exclusive' >Thread-498::DEBUG::2014-04-11 23:00:50,476::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8`ReqID=`2db1388a-4df6-4f17-bf71-271537b05f2e`::Granted request >Thread-503::DEBUG::2014-04-11 23:00:50,479::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`14c6b8c9-f764-4cbd-bdb2-30e053e94814`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain' >Thread-499::DEBUG::2014-04-11 23:00:50,481::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free. Now locking as 'exclusive' (1 active user) >Thread-503::DEBUG::2014-04-11 23:00:50,485::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' for lock type 'exclusive' >Thread-499::DEBUG::2014-04-11 23:00:50,489::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87`ReqID=`139ca213-17f3-4c3a-b8ea-63e7f5bf8511`::Granted request >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,496::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=79c99542d0e81c31ac775955c32d527fa77a18c6'] >Thread-500::DEBUG::2014-04-11 23:00:50,498::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free. Now locking as 'exclusive' (1 active user) >Thread-499::DEBUG::2014-04-11 23:00:50,504::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `e2ffd939-380e-4fbb-8a23-5a0e69c84a87` is not a data domain it is an 2 domain, not upgrading >Thread-500::DEBUG::2014-04-11 23:00:50,504::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce`ReqID=`26a15518-3c6f-4405-987e-44a39be5a581`::Granted request >Thread-499::DEBUG::2014-04-11 23:00:50,510::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' >Thread-503::DEBUG::2014-04-11 23:00:50,511::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free. Now locking as 'exclusive' (1 active user) >Thread-503::DEBUG::2014-04-11 23:00:50,517::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.21619c8e-99ea-4813-be02-d708971e5393`ReqID=`14c6b8c9-f764-4cbd-bdb2-30e053e94814`::Granted request >Thread-499::DEBUG::2014-04-11 23:00:50,525::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users) >Thread-498::DEBUG::2014-04-11 23:00:50,526::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost3storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,99c3f3f8-7b13-44ec-9645-f350e4b453dd:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=423b56b3-3148-468c-8c28-4c96fddbc1c8', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7c657c4086b93fb5d45b8d6fa339b0cd6a379955'] >Thread-499::DEBUG::2014-04-11 23:00:50,530::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it. >Thread-499::DEBUG::2014-04-11 23:00:50,535::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records. >Thread-498::DEBUG::2014-04-11 23:00:50,542::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-499::DEBUG::2014-04-11 23:00:50,551::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' >Thread-498::DEBUG::2014-04-11 23:00:50,552::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-500::DEBUG::2014-04-11 23:00:50,554::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost4storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Regular', 'SDUUID=b7663d70-e658-41fa-b9f0-8da83c9eddce', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=40b512932606eb692f63319d122b937229315e40'] >Thread-498::DEBUG::2014-04-11 23:00:50,560::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-498::DEBUG::2014-04-11 23:00:50,568::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-500::DEBUG::2014-04-11 23:00:50,570::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' >Thread-503::DEBUG::2014-04-11 23:00:50,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost1storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=35', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=21619c8e-99ea-4813-be02-d708971e5393', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4ad5bb5ea7e670d5988f412a6fbf2a7483772bd2'] >Thread-499::DEBUG::2014-04-11 23:00:50,575::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' (0 active users) >Thread-499::DEBUG::2014-04-11 23:00:50,582::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87' is free, finding out if anyone is waiting for it. >Thread-498::DEBUG::2014-04-11 23:00:50,583::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' >Thread-499::DEBUG::2014-04-11 23:00:50,588::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e2ffd939-380e-4fbb-8a23-5a0e69c84a87', Clearing records. >Thread-503::DEBUG::2014-04-11 23:00:50,588::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' >Thread-500::DEBUG::2014-04-11 23:00:50,591::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users) >Thread-500::DEBUG::2014-04-11 23:00:50,596::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it. >Thread-500::DEBUG::2014-04-11 23:00:50,596::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records. >Thread-498::DEBUG::2014-04-11 23:00:50,597::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' (0 active users) >Thread-498::DEBUG::2014-04-11 23:00:50,600::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8' is free, finding out if anyone is waiting for it. >Thread-500::DEBUG::2014-04-11 23:00:50,600::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' >Thread-498::DEBUG::2014-04-11 23:00:50,603::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_423b56b3-3148-468c-8c28-4c96fddbc1c8', Clearing records. >Thread-503::DEBUG::2014-04-11 23:00:50,604::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' (0 active users) >Thread-503::DEBUG::2014-04-11 23:00:50,607::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it. >Thread-503::DEBUG::2014-04-11 23:00:50,607::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.21619c8e-99ea-4813-be02-d708971e5393', Clearing records. >Thread-500::DEBUG::2014-04-11 23:00:50,608::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' (0 active users) >Thread-500::DEBUG::2014-04-11 23:00:50,611::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce' is free, finding out if anyone is waiting for it. >Thread-503::DEBUG::2014-04-11 23:00:50,612::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' >Thread-500::DEBUG::2014-04-11 23:00:50,613::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b7663d70-e658-41fa-b9f0-8da83c9eddce', Clearing records. >Thread-503::DEBUG::2014-04-11 23:00:50,616::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' (0 active users) >Thread-503::DEBUG::2014-04-11 23:00:50,619::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393' is free, finding out if anyone is waiting for it. >Thread-503::DEBUG::2014-04-11 23:00:50,619::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_21619c8e-99ea-4813-be02-d708971e5393', Clearing records. >Thread-501::DEBUG::2014-04-11 23:00:50,674::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >Thread-501::WARNING::2014-04-11 23:00:50,676::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >Thread-501::ERROR::2014-04-11 23:00:50,709::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:50,711::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:50,713::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,715::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,716::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-502::DEBUG::2014-04-11 23:00:50,734::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-502::WARNING::2014-04-11 23:00:50,745::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-502::ERROR::2014-04-11 23:00:50,779::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,995::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >8be642d5-95c4-4237-9639-a2844d95ee69::WARNING::2014-04-11 23:00:50,997::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:50,998::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,039::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,040::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,042::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 3406665e-4adc-4fd4-aa1e-037547b29adb >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,042::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 3406665e-4adc-4fd4-aa1e-037547b29adb >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,043::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,044::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3406665e-4adc-4fd4-aa1e-037547b29adb' (cwd None) >Thread-14::DEBUG::2014-04-11 23:00:51,097::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61] >Thread-14::DEBUG::2014-04-11 23:00:51,099::task::595::TaskManager.Task::(_updateState) Task=`13eae5eb-8ce5-4e9f-a17d-f264be3a0d1b`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:51,099::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='8be642d5-95c4-4237-9639-a2844d95ee69', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:51,099::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 8be642d5-95c4-4237-9639-a2844d95ee69 >Thread-14::DEBUG::2014-04-11 23:00:51,100::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'} >Thread-14::INFO::2014-04-11 23:00:51,100::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}} >Thread-14::DEBUG::2014-04-11 23:00:51,101::task::1185::TaskManager.Task::(prepare) Task=`13eae5eb-8ce5-4e9f-a17d-f264be3a0d1b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}} >Thread-14::DEBUG::2014-04-11 23:00:51,101::task::595::TaskManager.Task::(_updateState) Task=`13eae5eb-8ce5-4e9f-a17d-f264be3a0d1b`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:51,101::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:51,102::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:51,102::task::990::TaskManager.Task::(_decref) Task=`13eae5eb-8ce5-4e9f-a17d-f264be3a0d1b`::ref 0 aborting False >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,311::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found\n Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb\n'; <rc> = 5 >8be642d5-95c4-4237-9639-a2844d95ee69::WARNING::2014-04-11 23:00:51,313::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "3406665e-4adc-4fd4-aa1e-037547b29adb" not found', ' Skipping volume group 3406665e-4adc-4fd4-aa1e-037547b29adb'] >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,313::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,346::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 3406665e-4adc-4fd4-aa1e-037547b29adb not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,347::sp::329::Storage.StoragePool::(startSpm) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm > self._updateDomainsRole() > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole > domain = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,348::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,348::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,349::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`add08c36-8317-4d8e-b71e-51d8b1644024`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,349::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,350::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,350::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`add08c36-8317-4d8e-b71e-51d8b1644024`::Granted request >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,351::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`44fb3b1f-717b-4b6e-b8c1-36139fb31497`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,351::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' for lock type 'exclusive' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,352::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free. Now locking as 'exclusive' (1 active user) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,352::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb`ReqID=`44fb3b1f-717b-4b6e-b8c1-36139fb31497`::Granted request >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,353::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,353::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' (0 active users) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,354::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb' is free, finding out if anyone is waiting for it. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,354::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_3406665e-4adc-4fd4-aa1e-037547b29adb', Clearing records. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,355::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`7a624b7c-4ff5-4e26-a1a7-3cea91d6779c`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,355::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' for lock type 'exclusive' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,355::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free. Now locking as 'exclusive' (1 active user) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,356::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48`ReqID=`7a624b7c-4ff5-4e26-a1a7-3cea91d6779c`::Granted request >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,356::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,357::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' (0 active users) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,357::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48' is free, finding out if anyone is waiting for it. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,357::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f3b51811-4a7f-43af-8633-322b3db23c48', Clearing records. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,358::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,358::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,359::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,359::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,360::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,361::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,361::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vhost2storage', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=38', 'POOL_DESCRIPTION=SuprTEK', 'POOL_DOMAINS=423b56b3-3148-468c-8c28-4c96fddbc1c8:Active,e2ffd939-380e-4fbb-8a23-5a0e69c84a87:Active,b7663d70-e658-41fa-b9f0-8da83c9eddce:Active,3406665e-4adc-4fd4-aa1e-037547b29adb:Active,f3b51811-4a7f-43af-8633-322b3db23c48:Active,a033c2ac-0d01-490c-9552-99ca53d6a64a:Active,21619c8e-99ea-4813-be02-d708971e5393:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=a106ab81-9d5f-49c1-aeaf-832a137b708c', 'REMOTE_PATH=ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'ROLE=Master', 'SDUUID=a033c2ac-0d01-490c-9552-99ca53d6a64a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7adecf3e93be9a7e51a680211b46869bc3b9a109'] >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,391::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >8be642d5-95c4-4237-9639-a2844d95ee69::INFO::2014-04-11 23:00:51,392::clusterlock::279::SANLock::(release) Releasing cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,396::clusterlock::289::SANLock::(release) Cluster lock for domain a033c2ac-0d01-490c-9552-99ca53d6a64a successfully released >8be642d5-95c4-4237-9639-a2844d95ee69::ERROR::2014-04-11 23:00:51,398::task::866::TaskManager.Task::(_setError) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/storage/task.py", line 334, in run > return self.cmd(*self.argslist, **self.argsdict) > File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm > self._updateDomainsRole() > File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper > return method(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole > domain = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('3406665e-4adc-4fd4-aa1e-037547b29adb',) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,400::task::885::TaskManager.Task::(_run) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Task._run: 8be642d5-95c4-4237-9639-a2844d95ee69 () {} failed - stopping task >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,400::task::1211::TaskManager.Task::(stop) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::stopping in state running (force False) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,401::task::990::TaskManager.Task::(_decref) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::ref 1 aborting True >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,401::task::916::TaskManager.Task::(_runJobs) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,401::task::990::TaskManager.Task::(_decref) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::ref 0 aborting True >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,402::task::925::TaskManager.Task::(_doAbort) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Task._doAbort: force False >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,402::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,402::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state running -> state aborting >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,403::task::550::TaskManager.Task::(__state_aborting) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::_aborting: recover policy auto >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,403::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state aborting -> state racquiring >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,403::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state racquiring -> state recovering >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,404::task::798::TaskManager.Task::(_recover) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::_recover >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,404::task::805::TaskManager.Task::(_recover) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::running recovery None >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,404::task::786::TaskManager.Task::(_recoverDone) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::Recover Done: state recovering >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,405::task::595::TaskManager.Task::(_updateState) Task=`8be642d5-95c4-4237-9639-a2844d95ee69`::moving from state recovering -> state recovered >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,405::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>} >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,405::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,406::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,406::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,406::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,407::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >8be642d5-95c4-4237-9639-a2844d95ee69::DEBUG::2014-04-11 23:00:51,407::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-14::DEBUG::2014-04-11 23:00:52,107::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61] >Thread-14::DEBUG::2014-04-11 23:00:52,108::task::595::TaskManager.Task::(_updateState) Task=`fc0e5f16-24ff-481a-a7db-1b16af7edc6d`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:52,108::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='8be642d5-95c4-4237-9639-a2844d95ee69', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:52,109::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 8be642d5-95c4-4237-9639-a2844d95ee69 >Thread-14::DEBUG::2014-04-11 23:00:52,109::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'} >Thread-14::INFO::2014-04-11 23:00:52,109::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}} >Thread-14::DEBUG::2014-04-11 23:00:52,109::task::1185::TaskManager.Task::(prepare) Task=`fc0e5f16-24ff-481a-a7db-1b16af7edc6d`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '8be642d5-95c4-4237-9639-a2844d95ee69'}} >Thread-14::DEBUG::2014-04-11 23:00:52,110::task::595::TaskManager.Task::(_updateState) Task=`fc0e5f16-24ff-481a-a7db-1b16af7edc6d`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:52,110::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:52,110::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:52,111::task::990::TaskManager.Task::(_decref) Task=`fc0e5f16-24ff-481a-a7db-1b16af7edc6d`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:52,121::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61] >Thread-14::DEBUG::2014-04-11 23:00:52,121::task::595::TaskManager.Task::(_updateState) Task=`151fcf0b-de1c-4e8c-b8e1-3634038fa8c1`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:52,122::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-14::INFO::2014-04-11 23:00:52,136::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:52,137::task::1185::TaskManager.Task::(prepare) Task=`151fcf0b-de1c-4e8c-b8e1-3634038fa8c1`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-14::DEBUG::2014-04-11 23:00:52,137::task::595::TaskManager.Task::(_updateState) Task=`151fcf0b-de1c-4e8c-b8e1-3634038fa8c1`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:52,137::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:52,137::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:52,138::task::990::TaskManager.Task::(_decref) Task=`151fcf0b-de1c-4e8c-b8e1-3634038fa8c1`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:52,220::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [1ae41c61] >Thread-14::DEBUG::2014-04-11 23:00:52,220::task::595::TaskManager.Task::(_updateState) Task=`d9812f9c-8b12-4355-ad02-641ff85435ab`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:52,221::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='8be642d5-95c4-4237-9639-a2844d95ee69', spUUID=None, options=None) >Thread-14::DEBUG::2014-04-11 23:00:52,221::taskManager::161::TaskManager::(clearTask) Entry. taskID: 8be642d5-95c4-4237-9639-a2844d95ee69 >Thread-14::DEBUG::2014-04-11 23:00:52,221::taskManager::166::TaskManager::(clearTask) Return. >Thread-14::INFO::2014-04-11 23:00:52,222::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-14::DEBUG::2014-04-11 23:00:52,222::task::1185::TaskManager.Task::(prepare) Task=`d9812f9c-8b12-4355-ad02-641ff85435ab`::finished: None >Thread-14::DEBUG::2014-04-11 23:00:52,222::task::595::TaskManager.Task::(_updateState) Task=`d9812f9c-8b12-4355-ad02-641ff85435ab`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:52,222::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:52,223::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:52,223::task::990::TaskManager.Task::(_decref) Task=`d9812f9c-8b12-4355-ad02-641ff85435ab`::ref 0 aborting False >Thread-43::DEBUG::2014-04-11 23:00:52,764::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage/a033c2ac-0d01-490c-9552-99ca53d6a64a/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-43::DEBUG::2014-04-11 23:00:52,778::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000482084 s, 1.6 MB/s\n'; <rc> = 0 >Thread-24::DEBUG::2014-04-11 23:00:53,591::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso/e2ffd939-380e-4fbb-8a23-5a0e69c84a87/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-24::DEBUG::2014-04-11 23:00:53,605::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000340935 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2014-04-11 23:00:53,762::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage/423b56b3-3148-468c-8c28-4c96fddbc1c8/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2014-04-11 23:00:53,775::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n711 bytes (711 B) copied, 0.000528379 s, 1.3 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 23:00:54,084::task::595::TaskManager.Task::(_updateState) Task=`699bf6ac-cbff-4294-b4b1-66b159c7990a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:54,085::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 23:00:54,086::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000528379', 'lastCheck': '0.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340935', 'lastCheck': '0.5', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.0', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000564156', 'lastCheck': '9.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000482084', 'lastCheck': '1.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000302579', 'lastCheck': '9.9', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:54,086::task::1185::TaskManager.Task::(prepare) Task=`699bf6ac-cbff-4294-b4b1-66b159c7990a`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000528379', 'lastCheck': '0.3', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340935', 'lastCheck': '0.5', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.0', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000564156', 'lastCheck': '9.9', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.9', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000482084', 'lastCheck': '1.3', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000302579', 'lastCheck': '9.9', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:54,087::task::595::TaskManager.Task::(_updateState) Task=`699bf6ac-cbff-4294-b4b1-66b159c7990a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:54,087::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:54,087::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:54,088::task::990::TaskManager.Task::(_decref) Task=`699bf6ac-cbff-4294-b4b1-66b159c7990a`::ref 0 aborting False >Thread-29::DEBUG::2014-04-11 23:00:54,175::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage/b7663d70-e658-41fa-b9f0-8da83c9eddce/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-29::DEBUG::2014-04-11 23:00:54,188::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000517063 s, 1.5 MB/s\n'; <rc> = 0 >Thread-46::DEBUG::2014-04-11 23:00:54,227::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 23:00:54,240::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000417934 s, 1.8 MB/s\n'; <rc> = 0 >Thread-14::DEBUG::2014-04-11 23:00:57,224::task::595::TaskManager.Task::(_updateState) Task=`fd1bd315-844d-4f8a-a2a3-fe668e9efd70`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:57,225::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-14::INFO::2014-04-11 23:00:57,225::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000528379', 'lastCheck': '3.4', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340935', 'lastCheck': '3.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000517063', 'lastCheck': '3.0', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000482084', 'lastCheck': '4.4', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000417934', 'lastCheck': '3.0', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:57,226::task::1185::TaskManager.Task::(prepare) Task=`fd1bd315-844d-4f8a-a2a3-fe668e9efd70`::finished: {'423b56b3-3148-468c-8c28-4c96fddbc1c8': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000528379', 'lastCheck': '3.4', 'valid': True}, 'e2ffd939-380e-4fbb-8a23-5a0e69c84a87': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000340935', 'lastCheck': '3.6', 'valid': True}, '3406665e-4adc-4fd4-aa1e-037547b29adb': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.1', 'valid': False}, 'b7663d70-e658-41fa-b9f0-8da83c9eddce': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000517063', 'lastCheck': '3.0', 'valid': True}, 'f3b51811-4a7f-43af-8633-322b3db23c48': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.1', 'valid': False}, 'a033c2ac-0d01-490c-9552-99ca53d6a64a': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000482084', 'lastCheck': '4.4', 'valid': True}, '21619c8e-99ea-4813-be02-d708971e5393': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000417934', 'lastCheck': '3.0', 'valid': True}} >Thread-14::DEBUG::2014-04-11 23:00:57,226::task::595::TaskManager.Task::(_updateState) Task=`fd1bd315-844d-4f8a-a2a3-fe668e9efd70`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:00:57,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:00:57,227::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:00:57,227::task::990::TaskManager.Task::(_decref) Task=`fd1bd315-844d-4f8a-a2a3-fe668e9efd70`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:00:57,368::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [190fd93d] >Thread-14::DEBUG::2014-04-11 23:00:57,369::task::595::TaskManager.Task::(_updateState) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:00:57,369::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', hostID=1, remove=False, options=None) >Thread-14::DEBUG::2014-04-11 23:00:57,370::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d820e0af-8322-4826-a882-f82b53903bf9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1121' at 'disconnectStoragePool' >Thread-14::DEBUG::2014-04-11 23:00:57,371::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 23:00:57,371::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 23:00:57,371::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c`ReqID=`d820e0af-8322-4826-a882-f82b53903bf9`::Granted request >Thread-14::DEBUG::2014-04-11 23:00:57,372::task::827::TaskManager.Task::(resourceAcquired) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::_resourcesAcquired: Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c (exclusive) >Thread-14::DEBUG::2014-04-11 23:00:57,372::task::990::TaskManager.Task::(_decref) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::ref 1 aborting False >Thread-14::DEBUG::2014-04-11 23:00:57,373::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`bba216b4-bea8-46ac-8f46-68e7427cc35d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1129' at '_disconnectPool' >Thread-14::DEBUG::2014-04-11 23:00:57,373::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-14::DEBUG::2014-04-11 23:00:57,374::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-14::DEBUG::2014-04-11 23:00:57,374::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`bba216b4-bea8-46ac-8f46-68e7427cc35d`::Granted request >Thread-14::INFO::2014-04-11 23:00:57,375::sp::644::Storage.StoragePool::(disconnect) Disconnect from the storage pool a106ab81-9d5f-49c1-aeaf-832a137b708c >Thread-14::INFO::2014-04-11 23:00:57,375::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-22::DEBUG::2014-04-11 23:00:57,376::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 423b56b3-3148-468c-8c28-4c96fddbc1c8 >Thread-22::INFO::2014-04-11 23:00:57,376::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 (id: 1) >Thread-22::DEBUG::2014-04-11 23:00:59,377::clusterlock::217::SANLock::(releaseHostId) Host id for domain 423b56b3-3148-468c-8c28-4c96fddbc1c8 released successfully (id: 1) >Thread-14::INFO::2014-04-11 23:00:59,378::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-24::DEBUG::2014-04-11 23:00:59,379::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for e2ffd939-380e-4fbb-8a23-5a0e69c84a87 >Thread-14::INFO::2014-04-11 23:00:59,379::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-33::DEBUG::2014-04-11 23:00:59,380::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 3406665e-4adc-4fd4-aa1e-037547b29adb >Thread-14::INFO::2014-04-11 23:00:59,380::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-29::DEBUG::2014-04-11 23:00:59,381::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for b7663d70-e658-41fa-b9f0-8da83c9eddce >Thread-29::INFO::2014-04-11 23:00:59,381::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce (id: 1) >Thread-39::ERROR::2014-04-11 23:01:00,172::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::ERROR::2014-04-11 23:01:00,173::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 23:01:00,174::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-39::DEBUG::2014-04-11 23:01:00,175::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name f3b51811-4a7f-43af-8633-322b3db23c48' (cwd None) >Thread-39::DEBUG::2014-04-11 23:01:00,448::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found\n Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48\n'; <rc> = 5 >Thread-39::WARNING::2014-04-11 23:01:00,450::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "f3b51811-4a7f-43af-8633-322b3db23c48" not found', ' Skipping volume group f3b51811-4a7f-43af-8633-322b3db23c48'] >Thread-39::DEBUG::2014-04-11 23:01:00,450::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39::ERROR::2014-04-11 23:01:00,483::sdc::143::Storage.StorageDomainCache::(_findDomain) domain f3b51811-4a7f-43af-8633-322b3db23c48 not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-39::ERROR::2014-04-11 23:01:00,484::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain f3b51811-4a7f-43af-8633-322b3db23c48 monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('f3b51811-4a7f-43af-8633-322b3db23c48',) >Thread-29::DEBUG::2014-04-11 23:01:01,382::clusterlock::217::SANLock::(releaseHostId) Host id for domain b7663d70-e658-41fa-b9f0-8da83c9eddce released successfully (id: 1) >Thread-14::INFO::2014-04-11 23:01:01,383::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-39::DEBUG::2014-04-11 23:01:01,383::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for f3b51811-4a7f-43af-8633-322b3db23c48 >Thread-14::INFO::2014-04-11 23:01:01,384::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-43::DEBUG::2014-04-11 23:01:01,384::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for a033c2ac-0d01-490c-9552-99ca53d6a64a >Thread-43::INFO::2014-04-11 23:01:01,385::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a (id: 1) >Thread-15::DEBUG::2014-04-11 23:01:02,336::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-15::DEBUG::2014-04-11 23:01:02,337::task::595::TaskManager.Task::(_updateState) Task=`0373faeb-5195-4b7e-99d7-5966521c74c9`::moving from state init -> state preparing >Thread-15::INFO::2014-04-11 23:01:02,338::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', options=None) >Thread-15::INFO::2014-04-11 23:01:02,355::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-15::DEBUG::2014-04-11 23:01:02,355::task::1185::TaskManager.Task::(prepare) Task=`0373faeb-5195-4b7e-99d7-5966521c74c9`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-15::DEBUG::2014-04-11 23:01:02,356::task::595::TaskManager.Task::(_updateState) Task=`0373faeb-5195-4b7e-99d7-5966521c74c9`::moving from state preparing -> state finished >Thread-15::DEBUG::2014-04-11 23:01:02,356::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15::DEBUG::2014-04-11 23:01:02,356::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15::DEBUG::2014-04-11 23:01:02,357::task::990::TaskManager.Task::(_decref) Task=`0373faeb-5195-4b7e-99d7-5966521c74c9`::ref 0 aborting False >Thread-15::DEBUG::2014-04-11 23:01:02,361::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] >Thread-15::DEBUG::2014-04-11 23:01:02,362::task::595::TaskManager.Task::(_updateState) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::moving from state init -> state preparing >Thread-15::INFO::2014-04-11 23:01:02,362::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-15::ERROR::2014-04-11 23:01:02,363::task::866::TaskManager.Task::(_setError) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 2115, in getAllTasksStatuses > allTasksStatus = sp.getAllTasksStatuses() > File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper > raise SecureError("Secured object is not in safe state") >SecureError: Secured object is not in safe state >Thread-15::DEBUG::2014-04-11 23:01:02,363::task::885::TaskManager.Task::(_run) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::Task._run: 610b39ba-e53e-464a-8cc8-72bf57fbf266 () {} failed - stopping task >Thread-15::DEBUG::2014-04-11 23:01:02,364::task::1211::TaskManager.Task::(stop) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::stopping in state preparing (force False) >Thread-15::DEBUG::2014-04-11 23:01:02,364::task::990::TaskManager.Task::(_decref) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::ref 1 aborting True >Thread-15::INFO::2014-04-11 23:01:02,364::task::1168::TaskManager.Task::(prepare) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100 >Thread-15::DEBUG::2014-04-11 23:01:02,365::task::1173::TaskManager.Task::(prepare) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::Prepare: aborted: Secured object is not in safe state >Thread-15::DEBUG::2014-04-11 23:01:02,365::task::990::TaskManager.Task::(_decref) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::ref 0 aborting True >Thread-15::DEBUG::2014-04-11 23:01:02,365::task::925::TaskManager.Task::(_doAbort) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::Task._doAbort: force False >Thread-15::DEBUG::2014-04-11 23:01:02,366::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15::DEBUG::2014-04-11 23:01:02,366::task::595::TaskManager.Task::(_updateState) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::moving from state preparing -> state aborting >Thread-15::DEBUG::2014-04-11 23:01:02,366::task::550::TaskManager.Task::(__state_aborting) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::_aborting: recover policy none >Thread-15::DEBUG::2014-04-11 23:01:02,366::task::595::TaskManager.Task::(_updateState) Task=`610b39ba-e53e-464a-8cc8-72bf57fbf266`::moving from state aborting -> state failed >Thread-15::DEBUG::2014-04-11 23:01:02,367::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15::DEBUG::2014-04-11 23:01:02,367::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15::ERROR::2014-04-11 23:01:02,367::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run > result = ctask.prepare(self.func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1176, in prepare > raise self.error >SecureError: Secured object is not in safe state >Thread-46::DEBUG::2014-04-11 23:01:04,269::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage/21619c8e-99ea-4813-be02-d708971e5393/dom_md/metadata bs=4096 count=1' (cwd None) >Thread-46::DEBUG::2014-04-11 23:01:04,282::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n755 bytes (755 B) copied, 0.000376024 s, 2.0 MB/s\n'; <rc> = 0 >Thread-43::DEBUG::2014-04-11 23:01:04,386::clusterlock::217::SANLock::(releaseHostId) Host id for domain a033c2ac-0d01-490c-9552-99ca53d6a64a released successfully (id: 1) >Thread-14::INFO::2014-04-11 23:01:04,386::domainMonitor::113::Storage.DomainMonitor::(stopMonitoring) Stop monitoring 21619c8e-99ea-4813-be02-d708971e5393 >Thread-46::DEBUG::2014-04-11 23:01:04,387::domainMonitor::175::Storage.DomainMonitorThread::(_monitorLoop) Stopping domain monitor for 21619c8e-99ea-4813-be02-d708971e5393 >Thread-46::INFO::2014-04-11 23:01:04,387::clusterlock::207::SANLock::(releaseHostId) Releasing host id for domain 21619c8e-99ea-4813-be02-d708971e5393 (id: 1) >Thread-46::DEBUG::2014-04-11 23:01:06,388::clusterlock::217::SANLock::(releaseHostId) Host id for domain 21619c8e-99ea-4813-be02-d708971e5393 released successfully (id: 1) >Thread-14::DEBUG::2014-04-11 23:01:06,390::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-14::DEBUG::2014-04-11 23:01:06,390::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-14::DEBUG::2014-04-11 23:01:06,391::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 23:01:06,391::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-14::INFO::2014-04-11 23:01:06,392::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: True >Thread-14::DEBUG::2014-04-11 23:01:06,392::task::1185::TaskManager.Task::(prepare) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::finished: True >Thread-14::DEBUG::2014-04-11 23:01:06,392::task::595::TaskManager.Task::(_updateState) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:01:06,393::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c': < ResourceRef 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', isValid: 'True' obj: 'None'>} >Thread-14::DEBUG::2014-04-11 23:01:06,393::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:01:06,394::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' >Thread-14::DEBUG::2014-04-11 23:01:06,394::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' (0 active users) >Thread-14::DEBUG::2014-04-11 23:01:06,394::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c' is free, finding out if anyone is waiting for it. >Thread-14::DEBUG::2014-04-11 23:01:06,395::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a106ab81-9d5f-49c1-aeaf-832a137b708c', Clearing records. >Thread-14::DEBUG::2014-04-11 23:01:06,395::task::990::TaskManager.Task::(_decref) Task=`7748e2ae-202c-44ef-aa05-35dd2d80d15a`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:01:06,473::BindingXMLRPC::251::vds::(wrapper) client [172.16.10.1] flowID [159e2882] >Thread-14::DEBUG::2014-04-11 23:01:06,473::task::595::TaskManager.Task::(_updateState) Task=`c4b2385a-71b2-410b-b35d-11508237367c`::moving from state init -> state preparing >Thread-14::INFO::2014-04-11 23:01:06,474::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='a106ab81-9d5f-49c1-aeaf-832a137b708c', conList=[{'connection': 'ashtivh02.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c', 'port': ''}, {'connection': 'ashtivh03.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/exports', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54', 'port': ''}, {'connection': 'ashtivh04.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'deb06d0b-14c6-490e-8fbb-552861511109', 'port': ''}, {'connection': 'ashtistg01.suprtekstic.com:/storage/infrastructure', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760', 'port': ''}, {'connection': 'ashtivh01.suprtekstic.com:/var/lib/exports/storage', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac', 'port': ''}], options=None) >Thread-14::DEBUG::2014-04-11 23:01:06,475::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh02.suprtekstic.com:_var_lib_exports_storage' (cwd None) >Thread-14::DEBUG::2014-04-11 23:01:06,583::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh03.suprtekstic.com:_var_lib_exports_storage' (cwd None) >Thread-14::DEBUG::2014-04-11 23:01:06,640::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_exports' (cwd None) >Thread-14::ERROR::2014-04-11 23:01:06,664::hsm::2477::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2473, in disconnectStorageServer > conObj.disconnect() > File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect > return self._mountCon.disconnect() > File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect > self._mount.umount(True, True) > File "/usr/share/vdsm/storage/mount.py", line 253, in umount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (1, ';umount: /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_exports: not found\n') >Thread-14::DEBUG::2014-04-11 23:01:06,666::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh04.suprtekstic.com:_var_lib_exports_storage' (cwd None) >Thread-14::DEBUG::2014-04-11 23:01:06,709::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure' (cwd None) >Thread-14::ERROR::2014-04-11 23:01:06,746::hsm::2477::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2473, in disconnectStorageServer > conObj.disconnect() > File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect > return self._mountCon.disconnect() > File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect > self._mount.umount(True, True) > File "/usr/share/vdsm/storage/mount.py", line 253, in umount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 238, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (1, ';umount: /rhev/data-center/mnt/ashtistg01.suprtekstic.com:_storage_infrastructure: not found\n') >Thread-14::DEBUG::2014-04-11 23:01:06,747::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_iso' (cwd None) >Thread-14::DEBUG::2014-04-11 23:01:06,791::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/ashtivh01.suprtekstic.com:_var_lib_exports_storage' (cwd None) >Thread-14::DEBUG::2014-04-11 23:01:06,830::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-14::DEBUG::2014-04-11 23:01:06,830::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 23:01:06,831::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-14::DEBUG::2014-04-11 23:01:06,831::misc::764::SamplingMethod::(__call__) Got in to sampling method >Thread-14::DEBUG::2014-04-11 23:01:06,832::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-14::DEBUG::2014-04-11 23:01:06,832::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-14::DEBUG::2014-04-11 23:01:06,856::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2014-04-11 23:01:06,856::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) >Thread-14::DEBUG::2014-04-11 23:01:07,032::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2014-04-11 23:01:07,033::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 23:01:07,034::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 23:01:07,035::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 23:01:07,035::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 23:01:07,036::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-14::DEBUG::2014-04-11 23:01:07,036::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-14::DEBUG::2014-04-11 23:01:07,036::misc::772::SamplingMethod::(__call__) Returning last result >Thread-14::INFO::2014-04-11 23:01:07,037::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]} >Thread-14::DEBUG::2014-04-11 23:01:07,038::task::1185::TaskManager.Task::(prepare) Task=`c4b2385a-71b2-410b-b35d-11508237367c`::finished: {'statuslist': [{'status': 0, 'id': '3a714537-238b-4fbf-8362-da2f922b6e5c'}, {'status': 0, 'id': '523bf150-41d4-4c9c-9e5b-95b4452d3697'}, {'status': 477, 'id': 'dced514e-9d49-4df5-b32e-2d6a27acac54'}, {'status': 0, 'id': 'deb06d0b-14c6-490e-8fbb-552861511109'}, {'status': 477, 'id': 'eee1a342-05f6-4d3a-92fb-176201b5c73e'}, {'status': 0, 'id': 'f68d3d5c-3406-4c67-a6da-c642174a0760'}, {'status': 0, 'id': 'fd790dd2-b58f-406d-8fe0-fb951143f7ac'}]} >Thread-14::DEBUG::2014-04-11 23:01:07,038::task::595::TaskManager.Task::(_updateState) Task=`c4b2385a-71b2-410b-b35d-11508237367c`::moving from state preparing -> state finished >Thread-14::DEBUG::2014-04-11 23:01:07,039::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-14::DEBUG::2014-04-11 23:01:07,039::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-14::DEBUG::2014-04-11 23:01:07,039::task::990::TaskManager.Task::(_decref) Task=`c4b2385a-71b2-410b-b35d-11508237367c`::ref 0 aborting False >Thread-14::DEBUG::2014-04-11 23:02:27,065::API::1134::vds::(fenceNode) fenceNode(addr=172.16.10.6,port=,agent=ipmilan,user=root,passwd=XXXX,action=status,secure=,options=lanplus) >Thread-14::DEBUG::2014-04-11 23:02:27,140::API::1160::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=172.16.10.6 >login=root >action=status >passwd=XXXX >lanplus out Getting status of IPMI:172.16.10.6...Chassis power = On >Done > err >Thread-14::DEBUG::2014-04-11 23:02:27,148::API::1134::vds::(fenceNode) fenceNode(addr=172.16.10.6,port=,agent=ipmilan,user=root,passwd=XXXX,action=status,secure=,options=lanplus) >Thread-14::DEBUG::2014-04-11 23:02:27,252::API::1160::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=172.16.10.6 >login=root >action=status >passwd=XXXX >lanplus out Getting status of IPMI:172.16.10.6...Chassis power = On >Done > err
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1086951
: 885644 |
885645
|
885646